Confidence-driven workflow orchestrator for data labeling

ABSTRACT

One embodiment includes a computer-implemented data labeling platform. The platform provides a confidence-driven workflow (CDW) executable to receive and process labeling requests to label data items. The CDW comprises a set of executable labelers, each labeler in having a dynamically modeled confidence range. The execution path for processing a labeling request to label a data item is dynamically determined. Dynamically determining the execution path comprises dynamically determining a bounded number of candidate paths through the set of labelers using dynamically calculated cost and confidence metrics for the labelers in the set of labelers to estimate a probability of each candidate path to satisfy a set of constraints on cost and final result confidence, selecting a candidate path that minimizes cost for a specified confidence from the candidate paths as a selected path, executing a next labeler consultation according to the selected path to label the data item, and dynamically re-determining the remaining execution path using calculated results arising from executing the completed path steps.

RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 119(e)to U.S. Provisional Patent Application No. 62/884,512, entitled“Confidence-Driven Workflow Orchestrator,” filed Aug. 8, 2019, which ishereby fully incorporated by reference herein for all purposes.

TECHNICAL FIELD

Embodiments relate to computer systems and computer implemented methodsfor labeling data. Even more particularly, embodiments relate to systemsand methods for labeling data using a dynamically determined executionpath through one or more labelers.

BACKGROUND

Machine learning (ML) techniques enable a machine to learn how toautomatically and accurately make predictions based on historicalobservation. Training an ML algorithm involves feeding the ML algorithmwith training data. For example, training an ML algorithm forclassification involves feeding the ML algorithm to build an ML model.For example, training an ML algorithm to classify data may involvetraining the ML algorithm with training data to build an ML model formapping an input space to labels in a discrete label set. The accuracyof an ML model often depends on the quantity and quality of the trainingdata used to build the ML model.

An entire industry has developed around the preparation and labeling oftraining data. A number of companies provide platforms through whichexample data is distributed to human users for manual labelling. Thecustomer may be charged for the labeling services based on the humanexpertise required to label the data, the number of rounds of humanreview used to ensure the accuracy of the labelled data and otherfactors. The need for people to label the training data can havesignificant costs, both in terms of time and money. A new paradigm forlabeling data is therefore required.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the disclosure. It should be notedthat the features illustrated in the drawings are not necessarily drawnto scale. A more complete understanding of the disclosure and theadvantages thereof may be acquired by referring to the followingdescription, taken in conjunction with the accompanying drawings inwhich like reference numbers indicate like features and wherein:

FIG. 1 is a diagrammatic representation of one embodiment of a labelingenvironment;

FIG. 2 is a diagrammatic representation of one embodiment of a labeler;

FIG. 3 is a diagrammatic representation of one embodiment of an internalstructure of a labeler;

FIG. 4 is a diagrammatic representation of one embodiment of processingusing a human labeler;

FIG. 5 is a diagrammatic representation of one embodiment of a machinelearning (ML) labeler;

FIG. 6 is a diagrammatic representation of one embodiment of an MLlabeler in more detail;

FIG. 7 is a diagrammatic embodiment of one embodiment of aconfidence-driven workflow (CDW);

FIG. 8A is a diagrammatic representation of a first path through oneembodiment of a CDW and example confidence estimates;

FIG. 8B is a diagrammatic representation of a second path through oneembodiment of a CDW and example confidence estimates;

FIG. 8C is a diagrammatic representation of a third path through oneembodiment of a CDW and example confidence estimates;

FIG. 8D is a diagrammatic representation of a fourth path through oneembodiment of a CDW and example confidence estimates;

FIG. 9 is a flow chart illustrating one embodiment of determiningquality metric data;

FIG. 10 is a flow chart illustrating one embodiment of determining aconfidence estimate for a labeled result;

FIG. 11A and FIG. 11B are a flow chart illustrating one embodiment of amethod for dynamic path selection;

FIG. 12A is a diagrammatic representation of a single node path;

FIG. 12B is a diagrammatic representation of further searching a paththrough a CDW;

FIG. 12C is a diagrammatic representation of additional example pathsthrough a CDW;

FIG. 13 is a flow chart of one embodiment of processing by a labeler;

FIG. 14 is a diagrammatic representation of an example networkenvironment.

SUMMARY

As mentioned above, data labeling often relies on human specialists tolabel data. However, such data labeling is time consuming and expensive.Embodiments described herein provide mechanisms that can combine machinelearning based data labeling with human specialist data labeling. As themachine learning component becomes more accurate, the labeling platformcan automatically begin relying on the machine learning component moreheavily, for example routing requests to human specialists when themachine learning component produces a low confidence result.

In accordance with one aspect of the present disclosure, aconfidence-driven workflow (CDW) is provided. A CDW encapsulates acollection of labelers which are consulted in sequence, and theirindividual results are incorporated into an overall result, until aconfigured confidence threshold for an overall result is reached. Forexample, the collection of labelers can include machine learninglabelers, and human labelers, or combinations thereof. The constituentlabelers are not directly linked, and the execution path is dynamicallydetermined based, for example, on confidence and cost constraintconfiguration. The order of consultation generally proceeds from leastexpensive labeler to most expensive labeler. In some cases, a givenconstituent labeler may be consulted more than once in the executionpath.

The labelers in a workflow may act as interfaces to labeler instancesthat are continuously monitored and scored based on the results producedby the labeler instances. The scores for the labeler instances behind alabeler can be used to dynamically determine a confidence range for thelabeler and the confidence ranges for the labelers can be used indynamic path determination. More particularly, the dynamic pathdetermination mechanism can use the dynamically modeled confidenceranges for the labelers in the CDW to determine a priori confidenceestimates for one or more paths and identify candidate paths that arepredicted to meet cost and confidence constraints for a labelingrequest. The dynamic path determination mechanism can select a candidatepath based, for example, on minimizing cost or other criteria. The CDWexecutes the next labeler consultation for the labeling request based onthe selected path—for example, routes the labeling request to the nextlabeler based on the selected path.

As stated above, selection of a candidate path is based on expected costand impact on overall result confidence. The actual cost and impact onoverall result confidence may not be known until the individual labelerhas been consulted, and its result obtained and incorporated into theoverall result by the CDW. According to one embodiment, if the labeledresult returned by a labeler does not match the cost and/or confidenceexpectations, the CDW can dynamically redetermine candidate paths. Thisredetermination can incorporate the accrued cost and overall resultconfidence estimate, the configured cost and confidence constraints, andthe most recent available cost and confidence models of the CDWconstituent labelers.

It will be appreciated that the expectations for any given point in anexecution path may be different than the overall all expectations forthe execution path. After a step in the path, the labeling platform willhave more information about the actual confidence and costs so far inthe execution path and a redetermination of candidate paths can beperformed help optimize the path from the current point in the executionpath forward, whether or not the expectations for the current point inthe path have been met. In some embodiment, then, the (re)determinationof candidate paths can occur for every step in the execution path,whether or not the expectations for that point have been met (e.g.,until the overall all expectations for the execution path are met).

Dynamic path determination can account for the fact that the confidencein individual labelers may change. For example, as more data is labeled,a machine learning labeler can be retrained, and the quality of themachine learning labeler goes up. Consequently, the dynamicallydetermined execution paths may increasingly terminate after a singleconsultation with the machine learning labeler, driving down thetemporal and monetary costs of labeling by reducing reliance on humanspecialists.

One embodiment includes a computer program product comprising anon-transitory, computer-readable medium storing thereon a set ofcomputer-executable instructions. The set of computer executableinstructions can include instructions for providing a CDW comprising aset of labelers. Each labeler in the set of labelers can compriseexecutable code and have a dynamically modeled confidence range. The setof computer instructions can further comprise instructions fordynamically determining an execution path for a labeling request tolabel a data item, wherein dynamically determining the execution pathcomprises dynamically determining a bounded number of candidate pathsthrough the set of labelers using dynamically calculated cost andconfidence metrics for the labelers in the set of labelers to estimate aprobability of each candidate path to satisfy a set of constraints oncost and a final result confidence. The set of computer instructions canfurther comprise instructions for candidate path from the candidatepaths as a selected path. The next labeler consultation can be executedaccording to the selected path. For example, the labeling request can berouted to a next labeler based on the selected path.

Embodiments can further include instructions for receiving a labeledresult from the selected labeler and determining a confidence estimatefor the labeled result. The labeled result output of the selectedlabeler may be incorporated into an overall result. The overall resultmay be output as the final result for the CDW if the confidence estimatemeets the constraint for the final result confidence. If the estimatedconfidence in the overall result does not meet the target confidencethreshold, the labeling request may be routed to a next labeler from theset of labelers according to a selected path.

In another embodiment, if the estimated confidence in the labeled resultoutput by the labeler does not meet the target confidence threshold, thenext labeler consultation can be redetermined. For example, a new set ofcandidate paths can be redetermined using, for example, the accrued costand result confidence estimate based on prior labeler consultations inthe execution path and the confidence metrics for the labelers in theset of labelers to estimate a probability of each candidate path tosatisfy a set of constraints on cost and final result confidence. Acandidate path can be selected from the candidate paths and a nextconsultation executed according to the selected path.

If the confidence estimate for the overall labeled result does not meetthe target confidence threshold, an exception that the labeling requestcannot be completed within the cost and confidence constraints can bereported. For example, an exception can be reported if the confidenceestimate for the overall labeled result does not meet the targetconfidence threshold and there is no viable path to meet the cost andfinal result confidence constraints.

Embodiments can include instructions for continually monitoring andscoring a plurality of labeler instances to generate labeler instancescores for the plurality of labeler instances. Scoring a labelerinstance may include determining an accuracy of the labeler instancebased on a correctness of a set labeled results produced by the labelerinstance.

Embodiments can include instructions for updating the dynamicallymodeled confidence range for each labeler in the set of labelers.According to one embodiment, updating the dynamically modeled confidencerange for a labeler from the set of labelers comprises determining a setof labeler instance scores associated with a pool of labeler instancesrepresented by the labeler and aggregating the set of labeler instancescores to generate the dynamically modeled confidence range for thelabeler.

Each labeler instance can have an associated labeler instance cost(e.g., costs in one or more dimensions) and each labeler can have anassociated labeler cost. According to one embodiment, the associatedlabeler cost for each labeler is a statistical description of theassociated labeler instance costs of the one or more labeler instancesrepresented by that labeler.

According to one embodiment, a labeler may route labeling requests tolabeler instances based on scores. A labeler can receive an individuallabeler confidence constraint for a labeling request; determine alabeler instance of the one or more labeler instances represented bythat labeler that has a score that meets the individual labelerconfidence constraint; and route the first labeling request to the firstlabeler instance.

These, and other, aspects of the disclosure will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. It should be understood,however, that the following description, while indicating variousembodiments of the disclosure and numerous specific details thereof, isgiven by way of illustration and not of limitation. Many substitutions,modifications, additions, or rearrangements may be made within the scopeof the disclosure without departing from the spirit thereof, and thedisclosure includes all such substitutions, modifications, additions, orrearrangements.

DETAILED DESCRIPTION

Embodiments and the various features and advantageous details thereofare explained more fully with reference to the non-limiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well-known starting materials,processing techniques, components and equipment are omitted so as not tounnecessarily obscure the embodiments in detail. It should beunderstood, however, that the detailed description and the specificexamples are given by way of illustration only and not by way oflimitation. Various substitutions, modifications, additions and/orrearrangements within the spirit and/or scope of the underlyinginventive concept will become apparent to those skilled in the art fromthis disclosure.

Embodiments of the present disclosure provide systems and methods for aconfidence-driven workflow (CDW). A CDW encapsulates a set ofconstituent labelers having various associated confidences and costs.The CDW dynamically determines an execution path through one or morelabelers of the CDW based, for example, on cost and confidenceconstraints. More particularly, in accordance with one aspect of thepresent disclosure, the CDW dynamically determines a cost-optimizedexecution path to meet a target confidence threshold for a labelingrequest.

FIG. 1 is a diagrammatic representation of one embodiment of anenvironment 100 for labeling data. In the illustrated embodiment,environment 100 comprises a labeling platform system 102 coupled throughnetwork 105 to various computing devices. Labeling platform system 102provides a labeling platform 104 for labeling data. Network 105comprises, for example, a wireless or wireline communication network,the Internet or wide area network (WAN), a local area network (LAN), orany other type of communications link.

Labeling platform 104 executes on a computer—for example one or moreservers—with one or more processing units (CPUs, GPUs, and/or otherprocessing units) executing instructions embodied on one or morecomputer readable media where the instructions are configured to performat least some of the functionality associated with embodiments of thepresent invention. These applications may include one or moreapplications (instructions embodied on a computer readable media) one ormore interfaces 106 utilized by labeling platform 104 to gather datafrom or provide data to ML platform systems 130, human labeler computersystems 140, client computer systems 150, or other computer systems. Itwill be understood that the particular interface 106 utilized in a givencontext may depend on the functionality being implemented by labelingplatform 104, the type of network 105 utilized to communicate with anyparticular entity, the type of data to be obtained or presented, thetime interval at which data is obtained from the entities, the types ofsystems utilized at the various entities, etc. Thus, these interfacesmay include, for example web pages, web services, a data entry ordatabase application to which data can be entered or otherwise accessedby an operator, APIs, libraries or other type of interface which it isdesired to be utilized in a particular context.

Labeling platform 104 may comprise a number of services 108 forconfiguration, receiving input data to be labeled, outputting labeleddata, executing labelers, implementing confidence driven workflows(CDW), scoring labelers, dispatching tasks and/or implementing otherfunctionality. Labeling platform 104 further includes labeler core logic111 for multiple types of labelers and conditioning components 112 forvarious types of data conditioning. As discussed below, labeler corelogic 111 can be combined with conditioning components 112 to createlabelers 110.

Labeling platform 104 utilizes a data store 116 operable to storeobtained data, processed data determined during operation, andrules/models that may be applied to obtained data or processed data togenerate further processed data. Data store 116 may comprise one or moredatabases, file systems, combinations thereof or other data stores. Inone embodiment, data store 116 includes configuration data 117, whichmay include a wide variety of configuration data, including but notlimited to configuration data for configuring a directed graph, labelers110, and other aspects of labeling platform 104. Labeling platform 104also stores data to persist labelers 110 (labeler definitions 118), datato persist machine learning (ML) models 120 (ML model data 119),training data 122 used to train ML models 120, unlabeled data 124 to belabeled, quality metrics (QM) data 128 (e.g., confidence data) and otherdata.

Labeling platform 104 can distribute data to human users to be labeledand receive labeling results. To this end, environment 100 alsocomprises human labeler computer systems 140 that provide userinterfaces (UI) to present data to be labeled to human users and receiveinputs indicating the labels selected for the data by the human users.

Labeling platform 104 may also leverage ML models 120 to label data.Labeling platform 104 may implement its own ML platform or leverageexternal or third-party ML platforms, such as commercially available MLplatforms hosted on ML platform systems 130. As such, data labelingenvironment 100 includes one or more ML platforms in which ML models 120may be created, trained, and deployed. There are many platforms,frameworks, and algorithms available for ML model training andinference. By way of example, but not limitation, an ML model may betrained in a DOCKER container (e.g., a DOCKER container containinglibraries to train a model, or on a platform such as AMAZON SAGEMAKER,GOOGLE AUTOML, KUBEFLOW (SAGEMAKER from Amazon Technologies, Inc.,AUTOML from Google, DOCKER by Docker, Inc.). In addition, there arevarious model frameworks that can be used (e.g., TENSORFLOW by Google,PyTorch, and MXNet). Further there are many ML algorithms (e.g. K-Means,Logistic Regression, Support Vector Machines, Bayesian Algorithms,Perceptron, Convolutional Neural Networks). Labeling platform 104 cansend data to be labeled to one or more labeling platforms so that datacan be labeled by one or more ML models 120.

Client computer systems 150 provide interfaces to allow users, such asagents or customers of the entity that provides labeling platform system102, to create use cases and provide input data, such as submitunlabeled data 124 to be labeled. According to one embodiment, a usecase is a set of configuration information for configuring labelingplatform 104 to process unlabeled data 124. Even more particularly, ause case may be a configuration for a processing graph for processing ofunlabeled data 124. A use case may specify, for example, an endpoint foruploading records, an endpoint from which labelled records may bedownloaded, an endpoint from which exceptions may be downloaded, a listof output labels, characteristics of the unlabeled data (e.g., mediacharacteristics, such as size, format, color space), pipelines (e.g.,data validation and preparation pipelines), machine learningcharacteristics (e.g., ML model types, model layer configuration, activelearning configuration, training data configuration), confidence drivenworkflow configurations or other configurations. A confidence-drivenworkflow configuration may include a variety of configuration data for aconfidence-driven workflow including, but not limited to, a targetconfidence threshold, constituent labelers, whether each constituentlabeler are considered “open” or “blind” judgement labelers, the numberof times a labeling request may be resubmitted to the same labeler aspart of a CDW, human specialist workforces to use, task templates forhuman input, cost and quality constraints or other information. Labelingplatform 104 may support a wide array of use cases.

In operation, labeling platform 104 receives a set of data to be labeledand indicators of a set of labelers 110 available to use to label thedata. In accordance with one embodiment, labeling platform 104implements a use case to label the data. For example, the use case maypoint to a data source (such as a database, file, cloud computingcontainer, etc.) and specify configurations for labelers to use to labelthe data. Labeling platform 104 executes a directed graph of labelers110 (e.g., to implement the use case) to label the data. In some cases,the labelers are executed in a CDW for a use case to label the data andproduce labeled result data 126, where the workflow incorporates one ormore ML models and/or human users to label the data. The CDW may itselfbe implemented as a directed graph.

During execution of the workflow, the same data item to be labeled(e.g., image, video, word, document, or other discrete unit to belabeled) is sent to one or more labelers. The data item may be sent toone or more ML labeling platforms to be processed by one or more MLmodels 120. In addition, or in the alternative, the data item may besent to one or more human labeler computer systems 140 to be labeled byone or more human specialists. Based on the labels output for the dataitem by one or more labelers, the workflow can output a final labeledresult.

The CDW can be configured to dynamically determine the execution paththrough the one or more labelers to meet confidence and costconstraints. The dynamic path selected can depend on confidences andcosts of the one or more labelers and may be reevaluated after eachconsultation with a labeler in the path. The confidences in the labelerscan be continuously updated (e.g., by a quality management subsystem(QMS)).

A CDW implemented by labeling platform 104 can incorporate a lower costlabeler (e.g., ML labeler or human labeler) that initially has arelatively low accuracy for domain specific data being labeled. Overtime, the labeler may become more accurate, causing the confidence inthe labeler to increase. For example, an ML model behind an ML labelermay be retrained or human specialists behind a human labeler may becomemore accurate as they gain experience labeling a certain type of data.As the confidence in labeler increases, reliance on that labeler mayincrease. That is, the dynamic path selection may increasingly rely onpaths that route labeling requests to the low-cost labeler.

As an example, a CDW can include a generic ML image classifier in a CDWto label medical images, where the image classifier initially has a lowaccuracy for medical images. Over time, the image classifier can beretrained using medical images labeled by the CDW or other training datato become increasingly accurate. As the confidence in the imageclassifier's accuracy increases, CDW may dynamically change to rely moreheavily on the ML image classifier and reduce or eliminate humaninvolvement in labeling training data. Thus, the speed at which labelingplatform 104 can label data to a requisite degree of confidence mayincrease over time while the cost decreases.

In some embodiments, the CDW may be used to label domain specifictraining data that can then be used to train machine learning models.For example, a CDW may be used to label medical images and the labeledmedical images can be used as training data to train a machine learningmodel that is not part of the CDW or even labeling platform 104.

The basic building block of a workflow is a “labeler.” A labeler takesinput and enriches the input with one or more labels. Labelers(including labelers of different types) can be composed together intodirected graphs as needed, such that each individual labeler solves aportion of an overall classification problem, and the results areaggregated together to form the overall labeled output. The overalllabeling graph for a use case can be thought of abstractly as a singlelabeler, and each labeler may itself be implemented as a directed graph.There may be branches, merges, conditional logic, and loops in adirected graph. Each directed graph may include a fan-in to a singleoutput answer or exception per input element. The method of modeling thelabeling in such embodiments can be fractal. The labeling graphsimplemented for particular use cases may vary, with some graphs relyingexclusively on ML labelers and other graphs relying solely on humanlabelers.

Labeling platform 104 may include multiple types of labelers andmultiple labelers of each type. Example labelers include, but are notlimited to, executable code labelers, third-party hosted endpointlabelers, ML labelers, human labelers. As mentioned above, labelers maybe composed together. For example, labelers may be combined into CDWs. ACDW may also be considered a type of labeler.

FIG. 2 is a high-level block diagram of a labeler 200. Input is fed tolabeler 200 over an input pipe 202 and is passed through an inputconditioning pipeline if one is specified for the labeler. The labeledresult, which may be the result of running a (conditioned) labelingrequest through a labeler instance 203, is placed in an output pipe 204.The output labeled result can be passed through an output conditioningpipeline if one is specified for the labeler. Inputs that the labelerfails to label may be placed in an exception pipe 206. Some exceptionsmay be recoverable. Input pipe 202, output pipe 204, and exception pipe206 can pass both data and labeling flow control. Each of these pipescan have a configurable expected data schema.

An element of input data may be considered a labeling request, which cancomprise an element to be labeled or reference to the element to belabeled, such as an image or other data item to be labeled by thelabeler. The labeling request may have associated flow control data,such as constraints on allowable confidence and cost, a list of labelerinstances 203 acceptable to handle or not handle the request or otherassociated flow control information to control how the labeler 200handles the request.

According to one embodiment, the labeled result output by labeler 200 onoutput pipe 204 includes the data item or reference to the data itemlabeled and at least one label. The label output by labeler 200 may havemany forms, such as, but not limited to: a value output based on aregression model, a class label, a bounding box around an object in animage, a string of words that characterize/describe the input (e.g.,“alt text” for images), an identification of segmentation (e.g.,“chunking” a sentence into subject and predicate). In some cases,labeler 200 may also output a self-reported confidence measure for alabel. Labeler 200 may also output various other information associatedwith the labeled result, such as the labeler instance that processed thelabeling request.

A labeling request can be thought of as a question. For example,inputting an image to a labeler adapted for detecting tumors in imagesmay be thought of as the question “does this image include a tumor?” Alabeled result can be thought of as a judgement or an answer to aquestion. For example, in response to an image input into a labeleradapted to detect tumors in medical images, the labeler may tag an imageas “tumor” if a tumor is detected or as “no tumor,” indicating that notumor was detected, either of which is an answer to the question “doesthis image include a tumor?”

One embodiment of the general internal structure of a labeler 300 isillustrated in FIG. 3 . A labeler may be considered a wrapper onexecutable code. In some cases, the executable code may call out tothird party hosted endpoints. Configuration can specify the endpoints touse, authentication information, and other configuration information toallow the labeler to use the endpoint. In the illustrated embodiment,the labeler's kernel core logic 302 is surrounded by a conditioninglayer 304, which translates input/output data from an external domain tothe kernel's native data domain. As will be appreciated, differentlabelers may have different kernel core logic 302 and conditioninglayers 304. Some types of labelers may include additional layers.

Each human labeler and ML labeler may be considered an interface to apool of one or more labeler instances behind it. A labeler is in chargeof routing labelling requests to specific labeler instances within itspool. For a human labeler, the labeler instances are individual humansworking through a user interface (e.g., human specialists). For an MLlabeler, the labeler instances are ML models deployed in modelplatforms. The kernel core logic 302 of a human labeler is configured todistribute labeling requests out to individual human specialists whilethe kernel core logic 302 of an ML labeler is configured to leverage anML model to label data. The labeler instances behind a labeler may havedifferent performance characteristics modeled by the labeling platform,including but not limited to, confidence metrics and costs (e.g., timecosts and monetary costs).

Translation by conditioning layer 304 may be required because the datadomain external to the kernel core logic 302 may be different than thekernel's data domain. In one embodiment, for example, the external datadomain may be use-case specific and technology agnostic, while thekernel's data domain may be technology-specific and use-case agnostic.The conditioning layer 304 may also perform validation on inbound data.For example, for one use case, a solid black image may be valid fortraining/inferring, while for other use cases, it may not. If it is not,the conditioning layer 304 may, for example, include a filter to removesolid black images. Alternatively, it might reject such input and issuean exception output.

The conditioning layer 304 of a labeler may include input conditioning,successful output conditioning, and exception output conditioning. Eachof these can be constructed by arranging conditioning components intopipelines. Conditioning components perform operations such as datatransformation, filtering, and (dis)aggregation. Similar to labelers,the conditioning component may have data input pipes, data output pipes,and exception pipes.

As mentioned above, some examples of labelers include, but are notlimited to, executable code labelers, third-party hosted endpointlabelers, ML labelers, human labelers, and confidence driven workflows(CDWs).

Executable code labelers package up executable code with configurableparameters to be used as executable code labelers. The configuration foran executable code labeler includes any configuration informationrelevant to the executable code of the labeler. Other than the genericconfiguration information that is common to all labelers, theconfiguration for an executable labeler will be specific to the code.Examples of things that could be configured include, but are not limitedto: S3 bucket prefix, desired frame rate, email address to be notified,batch size.

A third-party hosted endpoint labeler can be considered a special caseof an executable code labeler, where the executable code calls out to athird-party hosted endpoint. The configuration of the third-party hostedendpoint can specify which endpoint to hit (e.g., endpoint URL), authcredentials, timeout, etc.

A human labeler acts as a gateway to a human specialist workforce. Ahuman labeler may encapsulate a collection of human specialists withsimilar characteristics (cost/competence/availability/etc.) as well asencapsulating the details of routing requests to the individual humansand routing their results back to the labeling system. Human labelerspackage the inbound labeling request with configured specialistselection rules and a task UI specification into a task.

FIG. 4 illustrates one embodiment of processing by a human labeler 400.In the illustrated embodiment, human labeler 400 receives a labelingrequest on input pipe 402 and outputs a labeled result on an output pipe404. Exceptions are output on exception pipe 406. Human labeler 400 mayinclude kernel core logic configured to distribute labeling requests outto individual human specialists and a conditioning layer to conditionone or more of labeling requests, labelled results, or exceptions.

Human labeler 400 is configured according to a workforce selectionconfiguration 410 and a task UI configuration 412. Workforce selectionconfiguration 410 provides criteria for selecting human specialists towhich a labeling request can be routed. Workforce selectionconfiguration 410 can include, for example, platform requirements,workforce requirements and individual specialist requirements. In someembodiments, labeling platform 104 can send tasks to human specialistsover various human specialist platforms (e.g., Amazon Mechanical Turkmarketplace and other platforms). Workforce selection configuration 410can thus specify the platform(s) over which tasks for the labeler can berouted. Human specialist platforms may have designated workforces(defined groups of human specialists). Workforce selection configuration410 can specify the defined groups of human specialists to which tasksfrom the labeler can be routed (i.e., groups of human labeler instancesto whom labeling tasks can be routed). If a workforce is declared in theworkforce selection configuration 410 for a use case, a human specialistmust be a member of that workforce for tasks for human labeler 400 to berouted to that human specialist. Workforce selection configuration 410may also specify criteria for the individual specialists to be routed atask for human labeler 400. By way of example, but not limitation,workforce selection configuration 410 can include a skill declarationthat indicates the skills and minimum skill scores that individualworkers (human specialists) must have to be routed labeling tasks fromthe labeler. A quality monitoring subsystem (QMS) may track skills/skillscores for individual human specialists.

Task UI configuration 412 specifies a task UI to use for a labeling taskand the options available in the UI. According to one embodiment, anumber of task templates can be defined for human labeler specialistswith each task template expressing a user interface to use forpresenting a labeling request to a human for labeling and receiving alabel assigned by the human to the labeling request. Task UIconfiguration 412 can specify which template to use and the labelingoptions to be made available in the task UI.

When human labeler 400 receives a labeling request, human labeler 400packages the labeling request with the workforce selection configuration410 and task UI template configuration 412 as a labeling task and sendsthe task to dispatcher service 409. Dispatcher service 409 is a highlyscalable long-lived service responsible for accepting tasks from manydifferent labelers and routing them to the appropriate endpoint forhuman specialist access to the task. Once a worker accepts a task,labeling platform 104 (e.g., dispatcher service 409) serves theconfigured browser-based task UI 420, then accepts the task result fromthe specialist and validates it before sending it back to the labeler.The same labeling request may be submitted multiple times to a singlehuman labeler. In some embodiments however, it is guaranteed that alabeling request is not presented to the same human specialist (humanlabeler instance) more than once.

Human-facing tasks can also support producing an exception result, witha reason for the exception. In some embodiments, human-facing tasks mayallow a specialist to self-assess their confidence in their answer. Assuch, a task result may include an associated self-resulted confidencemeasure.

FIG. 5 illustrates one embodiment of an ML labeler 500. The core logicof ML labeler 500 may implement an ML model 501 or connect to an MLframework to train or utilize ML model 501 in the ML framework. In theillustrated embodiment, ML labeler 500 receives a labeling request oninput pipe 502, runs the labeling request through an ML model 501—whichmay be considered a labeler instance—and outputs a labeled result onoutput pipe 504. As will be appreciated, ML model 501 may support thecapability to output a self-assessed confidence for a result. Thelabeled result output by ML labeler 500 may thus include an associatedself-assessed confidence for the result. Exceptions are output onexception pipe 506.

ML labeler 500 includes two additional input pipes, training data inputpipe 508 and quality metrics pipe 510. Training data input pipe 508 maybe used to receive training data for training (including retraining) ofML model 501. Quality metrics data is received over quality metrics pipe510. Quality metrics may be used to input quality metrics data for atraining flow or quality metrics data used, for example, in a CDW.Because the model used by the ML labeler can be retrained, ML labeler500 can learn over time to perform some or all of a use case.

Training data input pipe 508 and quality metrics pipe 510 can beconnected to the core logic of ML labeler 500 code (e.g., kernel corelogic 302 of ML labeler 500) similar to the input pipe as illustrated inFIG. 3 . ML labeler 500 may also include a conditioning layer tocondition one or more of labeling requests, training data, qualitymetrics data, labelled results, or exceptions.

At a high level, ML training and inference can be thought of as apipeline of five functional steps: input data acquisition, input dataconditioning, training, model deployment, and model inference. Accordingto one embodiment, the acquisition of unlabeled data for labeling andlabeled data for training is handled by labeling platform 104, asopposed to within the ML labeler 500 itself. By way of example, but notlimitation, the data may be passed in directly over an endpoint,streamed in via a queue like SQS or Kafka, or provided as a link to alocation in a blob store. The labeler can use simple standard librariesto access the data.

Data may be transformed to prepare the data for training and/orinference. Frequently some amount of transformation will be requiredfrom raw input data to trainable/inferable data. This may includevalidity checking, image manipulation, aggregation, etc. As would beappreciated by those in the art, the transformations can depend on therequirements of the ML model being trained or used for inference.

Training (and retraining) is the process by which conditioned trainingdata is converted into an executable model or a model is retrained. Theoutput of training is an ML model that represents the best modelcurrently producible given the available training data. It can be notedthat in some embodiments, such as embodiments utilizing ensembleapproaches, an ML labeler may use multiple models produced fromtraining.

Training data enters ML labeler 500 through its training data input pipe508. This pipe, according to one embodiment, transfers data only, notlabeling flow control. The schema of the training data input pipe may bethe same as the schema of output pipe 504. As such, training data mayneed conditioning in order to be consumable by the training process. Insome embodiments, training data accumulates in a repository, but may besubject to configurable data retention rules.

In some cases, end user-provided data or a publicly available datasetmay be used as a training dataset. New models can be trained asadditional training data becomes available. In addition, or in thealternative, training data can come from an “oracle” labeler (e.g., anoracle ML labeler or oracle human labeler). The output of the oraclelabeler is assumed to be correct, or at least the most correct to whichlabeling platform 104 has access for a use case.

Training data augmentation may be used to bolster and diversify thetraining data corpus by adding synthetic training data. This synthetictraining data can be based on applying various transforms to rawtraining data.

There are a variety of options for triggering training. The trigger maybe as simple as a certain number of training data records accumulating,or a certain percentage change therein. A training trigger may alsoincorporate input from a quality control subsystem. Time since lasttraining can also be considered.

Output labels from ML labeler 500 are the result of running aconditioned label request through a deployed ML model 501 to obtain aninferred answer. This inference may not be in a form that is directlyconsumable by the rest of the labeling graph (as specified by the schemaof output pipe 504), in which case the inference is passed through anoutput conditioning pipeline (e.g., in conditioning layer 304).According to one embodiment, the final labeled result output by MLlabeler 500, includes the input label request, the inferred label, and aself-reported confidence measure.

FIG. 6 is a diagrammatic representation of the functional components ofone embodiment of an ML labeler 600, which may be one example of an MLlabeler 500. According to one embodiment, ML labeler 600 is configuredaccording to an ML configuration that specifies a configuration of eachof the functional components.

FIG. 6 also illustrates example data labeling and training flows. In theembodiment of FIG. 6 , ML labeler 600 includes input pipe 602, outputpipe 604, training data input pipe 606 and quality metrics input pipe608. To simplify the diagram, the exception pipe is not shown in FIG. 6, but as will be appreciated, if any error condition is encountered inlabeler execution, it is signaled out on the exception pipe.

An ML labeler includes code to implement or utilize an ML model. In someembodiments, the ML labeler may be implemented as a wrapper for an MLmodel on an ML platform 650 running locally or on a remote ML platformsystem (e.g., an ML platform system 130). The ML labeler configurationcan specify an ML algorithm to use and, based on the ML algorithmspecified, labeling platform 104 configures the labeler with the code toconnect to the appropriate ML platform 650 to train and use thespecified ML algorithm.

An ML labeler can include a conditioning layer comprising a requestconditioning pipeline to condition input labeling requests, an inferenceconditioning pipeline to condition labeled results and a trainingrequest and label conditioning pipeline for conditioning training data.Each conditioning pipeline, if included, may comprise one or moreconditioning components. The ML labeler configuration can specify theconditioning components to be used for request conditioning, inferencede-conditioning and training and request conditioning and how thecomponents are configured (for example, can specify the size of image animage resizing component should resize to).

In the embodiment illustrated, ML labeler 600 includes a conditioninglayer comprising a training request conditioning pipeline 610 forconditioning training data to produce conditioned training data 612,which is used to train one or more ML models. The conditioning layerfurther includes request conditioning pipeline 632 to condition inputlabeling requests, and an inference conditioning pipeline 634 tocondition results. Labeling requests received on input pipe 602 areconditioned by request conditioning pipeline 632 and inferences(results) produced by a current active ML model are 620 are conditionedby interference (de)conditioning pipeline 634. Each conditioningpipeline, if included, may comprise one or more conditioning componentsas specified in the ML labeler's configuration.

ML labeler 600 includes training component 615 executable to train an MLalgorithm. Training component 615 may be configured to connect to theappropriate ML platform 650 to train an ML algorithm to create orretrain an ML model. The training component 615 includes an experimentcoordinator 616 that interfaces with ML platform 650 to train multiplechallenger models (e.g., using various hyperparameters or othermechanisms for training multiple candidate models known or developed inthe art) and a challenger model evaluator 618 that evaluates candidateML models against each other and the current active model to determinewhich should be the current active model for inferring answers tolabeling requests. The ML labeler configuration may further specifyhyper-parameter ranges and limits to be used during training. The outputis a champion ML model that represents the best model currentlyproducible given the available training data. The training component 615thus determines the ML model to use as the current active model forinferring answers to labeling requests.

The ML labeler configuration can specify training triggers, such thatwhen the training component 615 detects a training trigger, the trainingcomponent 615 initiates (re)training of the ML algorithm to determine acurrent active model. Training triggers may be based on, for example, anamount of training data received by the labeler, quality metricsreceived by the labeler, elapsed time, or other criteria.

In the illustrated embodiment, ML labeler 600 includes an activelearning record selector 630 to select records for active learning.Configuring active learning record selector 630 may include, forexample, specifying an active learning strategy (e.g., lowest accuracyor some other selection technique) and a batch size of records to passalong for further labeling and eventual use as training data for MLlabeler 600.

According to one embodiment, active learning record selector 630 selectsall unlabeled records (or some specified number thereof) for a use case(records that have not yet been labeled by the ML labeler) and has thoselabeled by the ML model 620. The ML model 620 evaluates its results(e.g., provides a confidence in its results). Active learning recordselector 630 evaluates the confidences in the results and forwards somesubset of the results to the other labelers in the graph and/or anoracle labeler for augmented labeling. These records then come back astraining data for the ML labeler (albeit potentially with a differentanswer as determined, for example, by a confidence-driven workflow).

The configuration for ML labeler 600 may include a general configurationand an ML labeler-type specific configuration. The ML labeler-typespecific configuration can include an ML algorithm configuration, atraining pipe configuration, and a training configuration. The MLalgorithm configuration specifies an ML algorithm or platform to use andother configuration for the ML algorithm or platform (layers to use,etc.). In some cases, a portion of the ML algorithm configuration may bespecific to the ML algorithm or platform. The training configuration caninclude an active learning configuration, hyper-parameter ranges,limits, and triggers. A portion of the training configuration may dependon the ML algorithm or platform declared. The ML labeler configurationcan also specify conditioning pipelines for the input, output, training,or exception pipes.

Turning now to confidence-driven workflows, ML labelers, human labelersand other labelers can be combined into a confidence-driven workflow(CDW). A CDW can thus be considered a labeler that encapsulates acollection of other labelers, and more particularly, a collection oflabelers of the same arity. The encapsulated labelers can be consultedin sequence and their individual results incorporated into an overallresult until a configured threshold confidence target for an overallresult is reached. A CDW can increase labeling result confidence bysubmitting the same labeling request to multiple constituent labelersand/or labeler instances. At a high level, multiple agreeing judgmentsabout the same labeling request can drive up confidence in the answer.On the other hand, a dissenting judgment can decrease confidence. A CDWmay include one or more ML labelers that can learn over time to performsome or all of a use case, reducing the reliance on human labeling, andtherefore driving down time and monetary cost to label data.

FIG. 7 is a diagrammatic representation of one embodiment of a CDW 700,which may be configured according to a CDW configuration. CDW 700receives input requests on input pipe 702 and outputs labeled results onoutput pipe 704. Exceptions are output on exception pipe 706. An elementof input received on input pipe 702 may be considered a labeling request(a question) and a labeled output may be considered an answer to thatquestion. A labeling request may be received as a workflow task withaccompanying task information, such as a task type description andconstraints. Examples of constraints include, but are not limited to,cost constraints in one or more dimensions (e.g., time limit, monetarylimit), target threshold confidence or other constraints. CDW 700receives quality metrics for constituent labelers from a qualitymonitoring subsystem (QMS) 750 on quality metrics pipe 708. Such qualitymetrics may include, for example, various scores, confidence estimates,confidence requirements that can be used by workflow orchestrator 710 todynamically determine a processing path through the labelers of CDW 700.

In the example embodiment illustrated, CDW 700 encapsulates ML labeler712, blind judgement human labeler 714, and open judgement human labeler716, though it should be appreciated that a CDW can encapsulate anynumber of labelers of various types. A labeler may be a gateway to a set(pool) of labeler instances (e.g., deployed ML models, humanspecialists) that all can service the same types of labeling request.For example, ML labeler 712 acts as an interface to a deployed ML model725 labeler instance, human labeler 714 acts as an interface to one poolof human specialists 727 and human labeler 716 acts as an interface to adifferent pool of human specialists 729. Multiple instances can sitbehind a single labeler where the labeler provides an interface to thelabeler instances. A labeler can thus have a set of labeler instancesbehind it and can be responsible for routing to specific labelerinstances within its pool (potentially through a dispatcher service asdiscussed above).

Each labeler instance may have an associated description. A labelerinstance description may be pulled from multiple sources such asconfiguration data 117, ML model data 119, QMS 750. A labeler instancedescription may include the task type description and a labeler instancescore. The task type description describes the type of tasks that can beperformed by the respective labeler instance. Each labeler instance mayhave a labeler instance score determined, for example, by QMS 750, andthat corresponds to the probability that the labeler instance willproduce an accurate label for a given task.

QMS 750 can continually score a labeler instance on how accurately thelabeler instance has performed tasks of the same task type in the past.For example, for a labeler instance that answered multiple instances ofthe question “does this image include a tumor?”, QMS 750 can score thelabeler instance based on how accurately the labeler instance answeredthe question over a number of images. According to one embodiment, thelabeler instance score is not specific to an answer but is aggregatedacross all the answers to the question that the labeler produced onprior task instances. In some embodiments, the score for the labelerinstance can be refined conditionally into answer-specific scores foreach label produced by the labeler. For example, for the question “doesthis image include a tumor”, the QMS 750 can score how often the labelerinstance was correct when it labeled images as “tumor” and score howoften the labeler instance was correct when it labeled images as “notumor.” The answer-specific scores can be used in determining confidenceestimates for an actual result output by the labeler instance

Human labeler instance scores and ML labeler instance scores can be tiedto specific labeling task types in the scoring system. In some cases, alabeler instance only produces labels for one type of task (e.g., on onetype of input data for one type of question) and the labeler instance'sscore may be tied to that task type. Some labeler instances, for examplehuman specialists, may be able to produce labels across a variety oftask types. The capability to produce labels for a task type may bereferred to as a skill and some labeler instances may support multipleskills. For a labeler instance that can produce labels for multiple tasktypes, QMS 750 may determine scores on the different task types as skillscores to differentiate the labeler instance's performance acrossdifferent labeling task types. Thus, a labeler instance, such as a humanspecialist, may have labeler instance scores for different labeling tasktypes. Alternatively, an aggregate score across the task types supportedby the labeler instance may be used.

A labeler instance description may also include costs in one or moredimensions. For example, a labeler instance may have an associatedresponse time (temporal cost) that is an estimate of how long it willtake that labeler instance to perform a task. As another example, alabeler instance may have an associated price (monetary cost) that is anestimate of the price for that labeler instance to perform a task. For alabeler instance with multiple skills, the description of the labelerinstance may include costs for each skill or a cost that applies acrossskills.

Each labeler has a labeler description. The labeler description for alabeler may be pulled from various sources such as from configurationdata 117, labeler definitions 118, QMS 750 or other sources. Accordingto one embodiment, a labeler description includes a task typedescription, associated costs (potentially in multiple dimensions) and alabeler score. The labeler description may also include otherinformation, such as a pool size, and descriptors.

A task type description describes the type of labeling request a labelercan service. That is, the task type description describes the type oftask that will be performed by the respective labeler instances of thatlabeler on input data. That task type description may be used forinitially configuring labelers available to workflow orchestrator 710.For example, for a particular labeling task, the workflow orchestrator710 may select from labelers suited for that type of task.

A labeler may also have a labeler score. A labeler's score correspondsto the probability that a labeler will produce an accurate label for agiven task. In some embodiments, the labeler's score may be astatistical description (e.g., statistical aggregate) of the labelerinstance scores for the labeler instances in that labeler's labelerinstance pool. For example, a labeler's score may be a confidence range(score range) for the labeler, such as min, max, mean and variance ofthe labeler instance scores for the labeler instances in that labeler'spool. A labeler may have different labeler scores for different tasktypes.

A labeler may have associated costs in one or more dimensions. In someembodiments, the labeler's cost in a dimension may be a statisticaldescription (e.g., a statistical aggregate) of the costs in thatdimension of the labeler instances in the labeler's pool. For example,the cost for a labeler in a dimension may be a mean and variance, a 95%confidence interval range, or other statistical representation of thecosts in that dimension of the labeler instances in that labeler's pool.

For example, each labeler instance behind a labeler may have a temporalcost (response time for that labeler instance to return a label result)and the temporal cost included in a labeler's description may bestatistical description of the temporal costs associated with thelabeler instances of that labeler, such as a mean and variance, a 95%confidence interval range, or other statistical representation of thetemporal costs of the labeler instances in that labeler's pool. Asanother example, each labeler instance in a labeler's pool may have anassociated monetary cost and the monetary cost included in a labeler'sdescription may be statistical description of the monetary costsassociated with the labeler instances in that labeler's pool, such as amean and variance, a 95% confidence interval range, or other statisticalrepresentation of the monetary costs of the labeler instances in thatlabeler's pool.

The pool size is the size of the pool of independent labeler instancesbehind the labeler, for repeat potential requests to the same labelerwith independent results.

Descriptors include information such as whether the labeler is an MLlabeler, human labeler. Descriptors may also include a variety of otherinformation, such as meta-data about instance pool, A/B condition fortesting, etc.

Workflow orchestrator 710 can be configured to determine or control howmuch visibility each labeler is permitted into accumulated judgementsfrom previous consultations. Labelers may also be specified as “open” or“blind” judgement (e.g., in the workflow configuration). Thus, in someembodiments, a labeler's description may include an indication ofwhether the labeler is a “blind judgement” or an “open judgementlabeler”.

A “blind judgement” labeler does not see the labels assigned for alabeling request by other labelers in the workflow. In the illustratedembodiment, for example, workflow orchestrator 710 does not provide thelabel determined by ML labeler 712 for a labeling request when providingthe same labeling request to human labeler 714 because human labeler 714is a “blind judgement” labeler. An “open judgement” labeler does see thelabels assigned by other labelers in the workflow. For example, in theillustrated embodiment, workflow orchestrator 710 passes the labelsdetermined by ML labeler 712 and blind judgement human labeler 714 forthe labeling request to human labeler 716 so that the human specialistcan see the previously determined labels. In general, allowing a labelerto see the result of previous consultation may enable it to perform itsown labeling faster, and therefore at lower cost. However, it also maybias the resulting judgment, which reduces its impact on the overallconfidence.

The constituent labelers of a CDW (e.g., labelers 712, 714, 716) are notdirectly linked. Instead, labeling platform 104 includes a workfloworchestrator 710 to dynamically determine an execution path forprocessing a labeling request (question) to produce a final labeledresult based on confidence and cost constraint configuration.

More particularly, according to one embodiment, workflow orchestrator710 uses the task information for the workflow task, the labelerdescriptions, possibly dynamic labeler characteristics such as cost,availability, and timeliness, and quality metrics from QMS 750 todynamically determine a path through the constituent labelers 712, 714,716 to produce a labeled output for the task which satisfies aconfigured target confidence threshold (a minimum confidence threshold).The path may also be selected to minimize costs in one or moredimensions.

As mentioned above, for a particular labeling task (input labelingrequest), the workflow orchestrator 710 may select from labelers suitedfor that type of task. Workflow orchestrator 710 uses the labeler scoresand costs associated with the labelers to determine one or more viablepaths through the labelers. When a labeler returns a result, aconfidence estimate for the result is determined using a score (or otherconfidence metrics) for the labeler instances in the path that haveprocessed the labeling request. According to one embodiment, once aviable path is determined, workflow orchestrator 710 may route thelabeling request through that path until a result reaches a thresholdconfidence target or the path is exhausted. In other embodiments, thepath may be re-evaluated and changed at any time based on the actualresults produced by each consultation.

Workflow orchestrator 710 routes the labeling request to the constituentlabelers by sending labeler tasks to the selected labelers, with thelabeler tasks including respective task information to be used by thatlabeler in processing the labeling request. A labeler can route alabeling request to a labeler instance as a labeling task, with thelabeling task including respective task information to be used by thelabeler instance to process the labeling request.

According to one embodiment, the order of consultation of constituentlabelers may proceed from least expensive labeler to most expensivelabeler, for example, in an attempt to reach the confidence thresholdtarget with the least cost. For example, if ML labeler 712 is the leastexpensive labeler, blind judgement human labeler 714 is more expensivebecause labeling requests to blind judgement human labeler 714 arerouted to human specialists who have a higher monetary cost based oncompensation for work performed, and human labeler 716 is the mostexpensive of the constituent labelers because labeling requests to humanlabeler 716 are routed to human specialists with higher expertise andcompensation levels than the human specialists associated with blindjudgement human labeler 714, then workflow orchestrator 710 may favorconsulting ML labeler 712 first, then human labeler 714, then humanlabeler 716.

Constituent labelers (e.g., labelers 712, 714, 716) are consulted untilthe configured target confidence threshold (e.g., as specified in a usecase) is reached or the path is exhausted (for example, there is noviable path to achieve the target confidence threshold). A givenconstituent labeler may be consulted more than once in the executionpath. If the same labeling request is submitted to the same humanlabeler (e.g., human labeler 714, human labeler 716) multiple times, thehuman labeler may be configured to route the request to a differentlabeler instance (human specialist), each with their own associatedquality metrics, each time. The workflow rules implemented by workfloworchestrator 710 can be configured to guarantee termination, so overallcost is bounded. As the quality of an ML labeler in a CDW goes up overtime, it is possible that most execution paths for a task type willterminate after a single consultation of the ML labeler.

For a labeling request, final result combiner 722 combines results fromthe multiple labelers in a CDW. In a simple case, there is one questionand all constituent labelers provide outputs corresponding to the samequestion, (e.g., does the image contain a tumor), and the judgments fromthe constituent labelers (e.g., tumor, no tumor, tumor) are stored in720 as they build up. The final result combiner 722 is configured torecognize the answer that reached the target confidence threshold andproduce a final result with that answer. For example, if the finalresult combiner determines that the answer “tumor” reached the targetconfidence threshold for a labeling request, the final results combiner722 can output the labeled result with the label “tumor.”

FIGS. 8A, 8B, 8C, 8D illustrate example execution paths usingconstituent labelers 712, 714, 716 of CDW 700. The “consultations axis”represents consultations executed using the constituent labelers and the“confidence” axis represents the confidence of the overall labelingresult that incorporates the individual labeling results obtained so farfor the labeling request. CDW can be configured dynamically determine anexecution path to optimize the confidence of the overall labeling resultthat incorporates the individual labeling results obtained so far forthe labeling request. Various strategies may be used by the CDW toincorporate the individual results into an overall result. One strategyis simply to evaluate the most recent individual result in isolation.Other strategies also be used. As discussed further below, the executionpath may be dynamically determined based on the scores and costsassociated with labelers 712, 714, 716.

Initially, ML labeler 712 may have a low confidence due to lack oftraining of ML model 725 (labeler instance). During this period, atypical labeler consultation sequence might be to route the labelingrequest as illustrated in FIG. 8A. In this example, the labeling requestis routed to ML labeler 712, which returns an answer for which aconfidence estimate 802 is determined. The labeling request is thenrouted to blind judgement human labeler 714, which produces an answerfor which a confidence estimate 804 is determined. As discussed below,confidence estimate 804 may be a complex granular confidence estimatebased on a simple granular confidence estimate determined for the answeroutput by ML labeler 712 and a simple granular confidence estimatedetermined for the answer output by human labeler 714. For example, ifthe answer output by human labeler 714 agrees with the answer output byML labeler 712, this may result in that answer being considered ahigh-confidence answer that exceeds the target confidence threshold.Thus, the answer (label) returned by human labeler 714 can be used forthe final labeled results of CDW 700.

In the example of FIG. 8B, the labeling request is routed to ML labeler712, which returns an answer for which a confidence estimate 806 isdetermined. The labeling request is then routed to blind judgement humanlabeler 714, which produces an answer for which a confidence estimate808 is determined. As the target confidence threshold has not beenreached, the labeling request can be routed to blind judgement humanlabeler 714 twice because there are labeler instances (human specialists727) remaining in the pool of human labeler 714 that have not yet beenconsulted for the labeling request. The confidence estimate 810 for theanswer produced by the second consultation with human labeler 714 mayincorporate confidence estimates 806, 808, however, exceeds the targetconfidence threshold. Thus, that answer can be used for the finallabeled results of CDW 700.

Results that meet the configured target confidence threshold (or othertraining data) can be used to retrain ML labeler 712. ML labeler 712 mayquickly become very accurate for the vast majority of labeling requeststhat it handles. Once this has been achieved, the ML labeler 712'sanswers may satisfy the configured target confidence threshold by itselfand many labeling requests to CDW 700 may be satisfied by ML labeler 712alone, reducing the time and cost associated with the use of humanspecialists. In FIG. 8C, for example, the processing path by constituentlabelers involves a single consultation by ML labeler 712, which canproduce an answer having a high confidence estimate 814 that exceeds thetarget confidence threshold for the labeling request. In this example,the answer returned by ML labeler 712 can be used for the final labeledresults of CDW 700 and no other labelers in the CDW are consulted.

FIG. 8D illustrates another example execution path. In this example, thelabeling request is routed to ML labeler 712, which returns an answerfor which a confidence estimate 820 is determined. The labeling requestis then routed to blind judgement human labeler 714, which produces ananswer for which a confidence estimate 822 is determined. As the targetconfidence threshold has not been reached, the labeling request isrouted to blind judgement human labeler 714 twice. The firstconsultation with human labeler 714 results in an answer for havingconfidence estimate 822 and the second consultation with human labeler714 results in an answer with a confidence estimate 824. The labelingrequest is routed to human labeler 716, which produces an answer havingconfidence estimate 826. In this example, the CDW cannot converge on anacceptably confident answer within configured execution limits. As such,the labeling attempt is considered a failure, and an exception isgenerated for the labeling result.

Returning to FIG. 7 , QMS 750 provides quality metrics that may be usedto dynamically determine the execution path and determine if a labelresult received from a constituent labeler (or agreed to by multiplelabelers) exceeds the configured target confidence threshold.

The probability that a given label result output by a labeler (e.g.,labeler 712, 714, 716) for a labeling request is accurate is referred toherein as a “confidence estimate.” That is, a “confidence estimate”refers to an estimate of confidence in the result (probability ofaccuracy). In other words, the “confidence estimate” is an amount ofconfidence the labeling system has in a result. This “confidenceestimate” (a value corresponding to the probability of accuracy) mayhave a (statistical) confidence interval around it.

According to one embodiment, QMS 750 performs a number of tasks tofacilitate the use of confidence estimates. QMS 750 provides confidenceestimates for a given label result to other parts of the architecture.Confidence estimates from QMS 750 for a labeled result can be used totrack the confidence of the result throughout the arc of a confidencedriven workflow. The confidence estimate for a label result can be used,for example, to determine if the labeled result has reached the targetthreshold confidence.

QMS 750 can continuously monitor and score labeler instances over timeto generate, maintain, and improve confidence estimates. Monitoring andscoring can be used by QMS 750 to generate the confidence estimates.

QMS 750 can predict what the confidence estimate will be for a givenlabeling task (before the labeling is done) under an array ofcircumstances where that task could be performed by different labelersor sets of labelers. The predicted confidence estimate describes thepredicted benefit from each set of potential labelers that workfloworchestrator 710 could select for a labeling request. The workfloworchestrator 710 can balance the predicted benefit against the costsassociated with each labeler to dynamically determine an execution path.

QMS 750 can identify the set of constraints on labelers required toachieve a desired confidence threshold for an overall result (which maybe composed of results from multiple labelers). For example, the QMS 750can back solve the confidence estimate equations relevant to aparticular labeling task so that, given a target confidence threshold,it identifies an individual labeler confidence constraint for eachparticipating labeler in a candidate sequence of labelers—that is, itidentifies the minimum confidence estimate for each labeler required toachieve the overall target confidence threshold. In some embodiments,the confidence estimate equation may have a closed-form solution and asimulation/sampling approach may be used to back-solve for theconfidence estimate constraints.

Confidence relates to the probability that a given label result isaccurate. As an indication of probability, confidence estimates rangebetween 0 and 1 according to some implementations. According to oneembodiment, QMS 750 estimates confidence on a per-judgement basis (atthe granularity of a single judgement/label). That is, QMS 750 maydetermine granular confidence estimates, where a granular confidenceestimate corresponds to a single label that cannot be further decomposedinto component labels. QMS 750 calculates composite confidence estimates(for results having more than one label) by combining granularconfidence estimates of the constituent labels.

Granular confidence estimates can be simple or complex. Simple granularconfidence estimates occur when there is only one labeler consulted onetime for a given label (one judgement, one label). Complex granularconfidence estimates are required when more than one judgement (almostalways from multiple, different labelers or labeler instances) impacts asingle label (multiple judgements, one label). Examples using complexgranular confidence estimates are when one labeler provides a judgementand another labeler reviews that judgement, or when multiple labelersare asked for the same judgement independently, and the final resultcorresponds to the answer that meets the target confidence threshold.

Simple granular confidence estimates are based on an individuallabeler's performance. The simple granular confidence estimate forresult produced by a labeler may be based on a score determined for thelabeler instance that processed the task or a self-reported confidenceprovided by the labeler instance. Complex granular confidence estimatescombine simple granular confidence estimates. According to oneembodiment, complex granular confidence estimates combine simplegranular confidence estimates in one of two ways (independent orconditional), each of which are configurable at the level of amathematical probability formula or ML estimation model for combininginto a single probability value.

Independent (Blind). The individual simple granular confidence estimatescontributing to a complex granular confidence estimate are treated asindependent if there is no interaction between the labelers. The orderin which the labelers are consulted does not impact their judgements,and the results of one labeler are not visible at any point to the otherlabelers. For example, in a labeling task that requires “multiple blindjudgements,” the labelers would be treated as independent. Note thatwhile the “blind judgement” labelers are independent of each other, theyare dependent on the actual labeling task/input, so a conditionalprobability formula can be used. The mathematical probability formulafor combining two multiple such blind judgements follows the“probability of disease given two independent tests” textbook paradigm.This formula can be extended to n number of labelers. For example,confidence estimate 804 of FIG. 8A for the judgement returned by humanlabeler 714 may be determined using a simple granular confidenceestimate determined for the judgement of ML labeler 712, a singlegranular confidence estimate for the judgement returned by human labeler714 (and implicitly whether the judgements agreed).

Conditional (Open). The individual simple granular confidence estimatescontributing to a complex granular confidence estimate are treated asconditional if there is interaction between the labelers (e.g., for opentasks in which later labelers can see the prior labeler's work). Theorder in which the labelers are consulted does matter, and the resultsfrom one labeler may be visible to other labelers. For example, alabeling task that requires one labeler to assess another labeler'soutput would be treated as conditional. In this case, one possiblemathematical probability formula for combining two judgements, onecontent and one review is straightforward conditional probability, theprobability that worker X (e.g., labeler instance) is correct given thatworker Y (e.g., labeler instance) accepts the answer isP(Xc|Ya)=(P(Ya|Xc)P(Xc))/P(Ya). This formula can also be extended to nnumber of labelers in the reviewer role.

Using the example formulas above, calculating complex granularconfidence estimates requires calculating probability values for priorand conditional probabilities above and beyond a simple granularconfidence estimate, P(Xc), the probability that a single labeler iscorrect for a single label. These additional probability values arecalculated based on counts/frequencies of actions observed on theplatform similar to how counts can be used to generate simple granularconfidence estimates, as described below.

It will be appreciated that the foregoing examples of complex granularconfidence estimates use probability models and are provided by way ofexample and not limitation and various methods of determining thecomplex granular confidence estimate may be used. For example,approaches may be employed that take into account the actual value ofthe answer (e.g., precision versus recall considerations, which intoaccount differences in false positives versus false negatives anddifferent likelihoods of providing one wrong answer versus a differentwrong answer when the true answer is x versus if it is y.) In someembodiments, logistic regression methods or other estimators may beapplied to determine complex granular confidence estimates rather thanclosed form probability equations. In some embodiments, an ML estimationmodel may be used for combining into a single probability value. By wayof example, but not limitation, logistic regression models can betrained for combining confidence estimates into a single probabilityvalue.

Some embodiments may implement multiple methods for determining complexgranular confidence estimates. It can be empirically determined whichmethod gives the best results for a particular workflow and that methodmay be used for the workflow.

When a result contains more than one label, QMS 750 can combine thegranular confidence estimates for the component labels to create acomposite confidence estimate for the overall result. For example, itmay be easier to label a video by splitting the video into frames andlabeling the individual frames. The confidence estimate for the videomay be a composite confidence estimate based on the confidence estimatesfor the individual frames. Consider, as another example, a use case forlocalizing and classifying retail products in an image where there arehundreds of possible product types. In such an embodiment, theconfidence estimate for the overall labeled result may be a composite ofthe confidence estimates for each product label.

The combination of confidence estimates to create a composite confidenceestimate is configurable at the level of a mathematical formula andwhich formula is selected may be dependent on the goals of the task andhow the confidence values will be interpreted. Some exampleconfigurations are listed below:

-   -   1) Product of the confidence values of all component parts.        Emphasizes importance of getting the entire set of labels being        correct without missing any. This is the classic probability of        co-occurrence of multiple independent events. Applies best to        truly independent component judgements where all parts must be        correct in order for the value proposition to be met.    -   2) The minimum confidence value over all the component parts        represents the confidence of the overall result. Emphasizes a        minimum threshold confidence for multi-part results. Applies        best to labeling compositions resulting from decomposition of        output label space or multi-judgement tasks (same as        decomposition of the output space, but a single labeler provides        all component judgements such as in a list of questions).    -   3) Alternately, a different minimum confidence value may be        configured/specified separately for each sub-component part, and        a distance metric can be specified based on how far each part is        from its minimum confidence. Emphasizes that no single number        can represent the confidence value for the entire task.

QMS 750 can determine confidence estimates from actual data observationson the platform (e.g., frequency counts of the event of interestcompared to total count of relevant events). Labeling platform 104 mayalso support heuristic and manually generated confidence estimationapproaches, as needed.

In general, a strong data-driven predictor of future performance is pastperformance. Platform can support any configurable algorithm forcalculating confidence estimates, based on, for example, event counts.The event counts of interest are counts of accurate and inaccuratelabels, which in some cases (e.g., categorical or binaryclassifications) can also be represented as a confusion matrix.

For example, if the platform knows how often a labeler instance hasgotten a particular kind of question right before, it can estimate thatthe labeler instance will get that question right in the future with thesame frequency (or proportion). In some embodiments, the platform canestimate the probability that a labeler instance will produce anaccurate label as a function of that labeler instance's history ofaccuracy. A more complex and robust model, such as a beta reputationapproach (see, Audun Jøsang, Roslan Ismail. The Beta Reputation System.15th Bled Electronic Commerce Conference e-Reality: Constructing thee-Economy Bled, Slovenia, Jun. 17-19, 2002, which is hereby fullyincorporated by reference herein) could be used.

QMS 750 continually measures and records the accuracy of labelerinstances' work and generates scores for each labeler instance, whichmay be stored in QM data 765. According to one embodiment, QMS 750assesses scores in generally the same way for human labeler instances orML labeler instances, by creating scoring actions for a set of labelsproduced by each labeler instance. For example, QMS 750 can determine ascore for each human represented by a human labeler and each ML model,including different versions of the same ML model trained on differentdata, represented by an ML labeler. For example, in the embodimentillustrated in FIG. 7 , QMS 750 can determine a score for ML model 725,each human specialist 727, and human specialist 729, and store thescores in QM data 765. The scores for labeler instances represented by alabeler may be aggregated into a score for the labeler. The score for alabeler may be, for example, a confidence range (score range).

QMS 750 may build up an initial score or scores for a labeler instancebased on a set of scorable (i.e., can be assessed by the platform asCORRECT or INCORRECT) tasks performed by that labeler instance, wherethe set of tasks may include repetitions of the same task type ondifferent inputs. QMS 750 aggregates scoring actions generated for thosetasks and determines an initial score or scores for the labelerinstance. For example, for a labeler instance that answered multipleinstances of the question “does this image include a tumor?”, QMS 750can score the labeler instance based on how accurately the labelerinstance answered the question over a number of images. According to oneembodiment, the labeler instance's score is not specific to an answerbut is an overall score aggregated across all the answers to thequestion that the labeler produced on prior task instances (e.g.,overall, how accurate the labeler instance was in answering thequestion). In some embodiments, the labeler instance score for thelabeler instance can be refined conditionally into answer-specificscores for each label produced by the labeler. For example, for aquestion, the QMS 750 can score how accurately a labeler instancelabeled images “tumor” and score how accurately the labeler instancelabeled images “no tumor.”

QMS 750 can use scores for individual labeler instances represented by ahuman or ML labeler to create an aggregate description of the labelerinstance scores represented by that labeler. QMS 750 may update scoresas data is labelled by the labeler instances.

FIG. 9 is a flow chart illustrating one embodiment of determiningquality metric data, including scoring labeler instances. The steps ofFIG. 9 may be embodied as computer-executable instructions on anon-transitory computer-readable medium. One or more steps of FIG. 9 maybe implemented by a labeling platform. By way of example, but notlimitation, one or more steps of FIG. 9 may be performed by QMS 750.

At step 902, results produced by labeler instances are stored (forexample as labeler results 755). It can be noted labeler results 755 canbe tracked at the level of the labeler instance that determined theresult. The labeler results may include, for example, the task type forwhich the result was generated, the identity of the labeler instancethat produced the result, the labeled result generated for the task, anyself-reported estimate of accuracy by the labeler instance, and otherdata.

QMS 750 can determine an initial labeler instance score for each labelerinstance based on a set of scorable tasks performed by the labelerinstance. In some cases, as determined at step 904, there may not beenough information (data) to calculate a score for a given labelerinstance for a given task type (e.g., because there are no gold answersto compare to, it is impractical to gather labels for scoring alone,etc.) In such cases, QMS 750 may use a bootstrapping method to calculatea score for a labeler instance (step 905). This initial score is anestimate, and—as a bootstrap—the initial estimate can be generated in avariety of heuristic ways that the labeling platform 104 can support asinput. Because heuristic approaches for creating a score generally donot have a solid foundation in quantifiable measurements, they can bereplaced with data-driven estimates as soon as enough data is availablefor statistical significance.

To provide some non-limiting examples, heuristic initial scores mayreflect estimates based on the following:

-   -   1) Labeler instance starts with a default score (for example,        0.5 indicating a 50/50 chance of getting an answer right or        wrong).    -   2) A score can be hard-coded for a particular labeler instance        based on a platform operator's prior experience with that        labeler instance.    -   3) In some cases, a human labeler instance's score on one task        may be used to bootstrap estimates of performance on another        task (e.g., bounding boxes for cats may be similar enough to        bounding boxes for cars, or fashion categorization may be        determined to correlate with skill on beauty product        categorization).    -   4) Characteristics of human labeler instances such as IQ tests        or other training quizzes may be used to create a heuristic        baseline that is scaled to other metrics deemed relevant by the        task designer.

According to one embodiment, the score for a labeler instanceself-corrects from the bootstrapped score through actual scoring overtime as the labeler instance performs more tasks that are scored. Asdata builds, the bootstrap heuristic models are replaced with datadriven models. In addition, the scoring approach itself can also beconfigured to discount older scored performance (for example, using betareputation decay).

If there is sufficient data to calculate a score for a labeler instance,processing can proceed to step 906. QMS 750, at step 906, createsscoring actions for the set of labels produced by the labeler instance.A scoring action is an assessment of whether an individual labeldetermined by the labeler instance for a data item is deemed CORRECT orINCORRECT. According to some embodiments, this is a binarydetermination, though other embodiments may incorporate varying(non-binary) degrees of correctness. There are various ways to make thisbinary CORRECT/INCORRECT determination, including, but not limited to:

-   -   1) Direct compare—an exact match to a known result. This        approach is applicable to: Binary yes/no, simple String match        (multiple choice or research tasks with one correct answer like        a known, fixed URL), deep JSON compare, fixed-form complex        result—this could be a deep data-structure compare for example.    -   2) Threshold—an acceptable degree of similarity to a known        result. Applicable to: polygon areas (IOU), transcriptions as        “close enough” with few missed words (measured by edit distance        for example), ordinal grades/valuations as +/−1, etc.    -   3) Subjective/Freeform. Applicable to: subjective        tagging/labeling, research tasks with open-ended acceptable        answers.

Direct and threshold comparisons can be programmatically calculatedusing independently derived results and comparison results. Subjectiveand freeform comparisons may require an assessment of the result itselfagainst some criteria and the CORRECT answer is therefore dependent onthe result itself (that is, a result can be judged to be CORRECT orINCORRECT but a closed-form CORRECT answer may not be available). In thesubjective/freeform case, a given label may be determined to be ACCEPTEDor REJECTED via a review/adjudication process.

In accordance with one embodiment of a scoring approach, determinationsfor direct and threshold comparisons are made against an “eventuallyaccepted” result that has been determined to be CORRECT. This could be agold record, the result of a workflow that gets a result to a highenough confidence, the result of a task that has been accepted by theend-user (the entity using the labeled results), and so forth. QMS 750may support scoring actions through various mechanisms. According to oneembodiment, QMS 750 compares the labeler results 755 (e.g., labelsoutput by labeler 712, 714, 716 for input elements/labeling requests) to“correct” labels 760 for the input elements/labeling requests, where thecorrect labels may be, for example: gold data (sometimes referred to as“ground truth” data), high confidence results (e.g., the final labelresults output by CDW 700 that exceed the confidence threshold) or otherlabels deemed to be correct. Scoring actions may also be recorded basedon, for example, workflow adjudications/overrides (review plus contentchanges), escalate/exception results, admin review, or end-userfeedback.

In any case, the labeled results produced by each labeler (e.g., MLlabeler 712, blind judgement human labeler 714 and open judgement humanlabeler 716) for a labeling request may be marked as CORRECT orINCORRECT, and the marked labeling results can be used to determine theaccuracy of the labeler instance that produced those results.

More than one type of scoring action can be used for a single label(e.g., gold and end-user feedback). Depending on the task (e.g., howambiguous cases might be, what type of comparisons are needed, thevolume of scoring actions available from various types, etc.), a scorefor a labeler instance can be configured as a weighted sum of scorescalculated from different scoring action sources (e.g., 50/50 gold andcustomer feedback), or configured to combine scoring actions of varioustypes into a single scoring formula (e.g., the alpha and beta parametersin a beta reputation model would include counts from both types ofscoring actions).

Some labeler instances only produce labels for one type of task (tasktype, e.g., on one type of input data for one type of question). Otherlabeler instances, in particular, human specialists, can produce labelsacross a variety of task types. Both human labeler instance scores andML labeler instance scores can be tied to specific labeling task typesin the scoring system (i.e., labeler instances can have different scoresrelevant to making different confidence estimates). In the case ofinstances that can produce labels for multiple task types, QMS 750 maydetermine scores on different task types as skill scores todifferentiate a labeler instance's performance across different labelingtask types. Thus, for example, a human labeler instance may have aplurality of scores, where each score corresponds to a differentlabeling task type or, alternately, an aggregate of related task types.

At step 908, QMS 750 can thus determine a labeler instance score(including, in some cases, multiple labeler instance scores) for eachlabeler instance reflecting overall reputation or reputation on somesubset of tasks or types of tasks, which corresponds to the probabilitythat a labeler instance will produce an accurate label for a given task,and which is calculated via a configurable mathematical algorithm using,for example, accuracy count data (including full confusion matrix countswhere applicable in classification tasks) of previously produced labels.These individual labeler instance scores can be aggregated at thelabeler level (e.g., ML labeler 712, blind judgement human labeler 714and open judgement human labeler 716) to generate a labeler score, whichcan be statistical description derived from the underlying labelerinstance scores.

At step 909, QMS can determine answer-specific scores for the labelerinstance. An answer-specific score represents how often the labelerinstance was correct in labeling data items with a particular label fora set of tasks (task instances of a task type). For example, for thequestion “does this image include a tumor”, the QMS 750 can score howoften the labeler instance was correct when it labeled images as “tumor”and score how often the labeler instance was accurate when it labeledimages as “no tumor.” The answer-specific scores can be used indetermining confidence estimates for an actual result output by thelabeler instance.

Many ML models report “confidence” (i.e., self-reported confidenceestimates) for the labels they produce. Humans can be asked self-reportaccuracy estimates as well, through the user interface, for each labelprovided. Because humans are typically bad at directly estimatingprobabilities accurately, the user experience for reporting confidencevalues may be subject to task design considerations and conversion toprobability estimates. As a simple case, when a human specialist istasked to provide a label, the human specialist can be asked for thelabel and a certainty value (1 to 5) for the label. In this example, aself-reported confidence estimate for the human can be calculated as amatrix of conditional probabilities using accuracy counts under eachcertainty condition (probability of correctness given that reportedcertainty is X) to determine a certainty-modified probability.

In any case, a labeler instance may self-report an estimate of theaccuracy of its answer—that is, a labeler instance may provide aself-reported confidence estimate. This self-reported estimate iscontextualized for each specific answer (label) on each specific input(data item labeled) and can therefore represent a more precise estimateof the probability of accuracy for a given result compared to othertypes of estimates. The self-reported estimates of accuracy by thelabeler can be included with the label result output by a labeler andstored in labeler results 755. With the prior label results marked ascorrect or incorrect, QMS 750 can determine and store a correlationbetween the self-reported probability of accuracy and the actualaccuracy for the labeler instance (step 914).

Steps 902-914 can be repeated for each labeler instance. At step 920,the scores for labeler instances represented by a common labeler can beaggregated for that labeler. For example, the scores for humanspecialists 727 can be aggregated to create a labeler score (forexample, a confidence range) for human labeler 714. As another example,the scores for ML models represented by ML labeler 712 can be aggregatedto determine a score for ML labeler 712.

QMS 750 may dynamically determine scores for labeler instances (e.g.,step 908). Thus, the confidence ranges of labelers can also bedynamically determined. In some embodiments, QMS 750 performs steps 906and 920 for a labeler—that is determines labeler instance scores for thelabelers represented by the labeler and creates a labeler score for thelabeler—in response to a request from a labeler. In other embodiments,QMS 750 continually determines labeler instance and labeler scores andprovides the scores when requested.

Moreover, after an initial score or scores are determined, QMS 750 cancontinue to score the labeler instances intermittently based oninference monitoring strategies, and thus the scores for the labelerinstances and the labelers can be updated. The number of scored actionsin the initial score(s) and the rate of intermittent scoring isconfigurable. QMS 750 may thus use scoring approaches where data aboutindividual labeler instances' accuracy is collected and updated as datais labeled on the platform.

FIG. 9 is provided by of example and not limitation. Various steps maybe repeated, steps may be performed in different orders, steps omitted,and additional or alternative steps performed.

In operation of CDW 700, the scores determined for labeler instances maybe used to determine confidence estimates for the answers output by eachlabeler. FIG. 10 is a flow chart illustrating one embodiment ofdetermining a granular confidence estimate for a result output by alabeler. The steps of FIG. 10 may be embodied as computer-executableinstructions on a non-transitory computer-readable medium. One or moresteps of FIG. 10 may be implemented by a labeling platform. By way ofexample, but not limitation, one or more steps of FIG. 10 may beperformed by QMS 750.

At step 1002, a labeler result is received. The labeler results mayinclude, for example, the task type for which the result was generated,the identity of the labeler instance that produced the result, thelabeled result output the labeler, any self-reported estimate ofaccuracy by the labeler instance, and other data. At step 1004, QMS 750determines the score for the labeler instance that determined theresult. For example, QMS 750 may access the labeler instance scoregenerated for the labeler instance at step 905 or step 908 and QMS 750uses the labeler instance score for the task type as the granularconfidence estimate for the labeled result output by the labeler (step1006). In other embodiments, QMS 750 may determine an answer-specificscore for the labeler instance (e.g., as determined at step 909), wherethe answer-specific score corresponds to the label output by the labelerinstance, and uses the answer-specific score as the granular confidenceestimate for the labeled result output by the labeler.

In some embodiments, when self-reported estimates are available, theymay be used to refine score-based estimates of accuracy. Note that undervarious cases, the QMS 750 may use the self-reported estimate directly,or use it to discount, increase, or otherwise modify a score-basedestimate. According to one embodiment, the QMS 750 uses the labeler'sself-reported confidence estimate for a label (e.g., as the simplegranular confidence estimate for the label) when:

-   -   1) The labeler instance's estimate of its confidence in the        label provided corresponds to a probability value; and    -   2) The correlation between the self-reported probabilities of        accuracy by the labeler instance and actual accuracy is high.

As discussed above, QMS 750 can track a correlation between theself-reported probability of accuracy and the actual accuracy for thelabeler instance. Thus, at step 1008, QMS 750 may access the correlationbetween the self-reported probability of accuracy and the actualaccuracy for the labeler instance for the task type and determine if thecorrelation is above a configurable threshold (step 1010). If thecorrelation meets the threshold, then the self-reported probabilityestimates can be used as the confidence estimate for the labeler result(step 1012). If the correlation does not meet the threshold, theconfidence estimate for the labeler instance reverts to the labelerinstance's score (e.g., as determined by initial and intermittentscoring) (step 1006).

FIG. 10 is provided by way of example and not limitation. Various stepsmay be repeated, steps may be performed in different orders, stepsomitted, and additional or alternative steps performed. Moreover, whileFIG. 10 describes an embodiment of determining a simple granularconfidence estimate, it will be appreciated that QMS 750 may determinecomplex granular confidence estimates when multiple labelers in a pathhave processed a labeling request. The complex granular confidenceestimates for an answer may be based on the answer-specific scoresdetermined for the labeler instances in the execution path, ifavailable, overall scores determined for the labeler instances for atask, self-assessed confidences of the labeler instances or combinationthereof. For example, the simple granular confidence estimate for eachlabeler instance may be determined as discussed in conjunction with FIG.10 , and the simple granular confidence estimates combined.

Turning now to path selection, labeling platform 104 may be configuredto route tasks to labelers based on any number of constraints that matchthe labelers' descriptions. Specific sequencing of labelers can bespecified to achieve predefined workflows. For example, a CDWconfiguration may specify that the workflow orchestrator first use aspecific ML Labeler with a pre-specified set of constraints that arepassed to the labeler for execution, followed by a specific humanlabeler, with a different pre-specified set of constraints. This allowspredefined workflows to be activated using the CDW in specificcircumstances (Testing, Crowd Curation, Validation). These predefinedworkflows offer only one path for the workflow orchestrator to route atask through specified labelers.

In addition, or in the alternative, CDW 700 can be configured with a setof labelers to use without predefining how those labelers are sequenced.Workflow orchestrator 710 determines viable execution paths through theconstituent labelers. An execution path through the constituent labelersis considered viable for a given task if it can satisfy all of thattask's constraints. Finding one or more viable execution paths does notguarantee a task's overall constraints will be satisfied because theactual confidence achieved after executing a planned path may differfrom the upfront prediction.

There may be more than one viable execution path through the constituentlabelers and workflow orchestrator 710 may be required to make routingdecisions. Workflow orchestrator 710 selects the labeler or sequence oflabelers to which to route based on the constraints. FIG. 11A and FIG.11B are a flow chart of one embodiment of dynamically determining a paththrough a set of constituent labelers. The steps of FIG. 11A and FIG.11B may be embodied as computer-executable instructions on anon-transitory computer-readable medium. One or more steps of FIG. 11Aand FIG. 11B may be implemented by a labeling platform. By way ofexample, but not limitation, one or more steps of FIG. 11A and FIG. 11Bmay be performed by workflow orchestrator 710 or QMS 750.

According to one embodiment, workflow orchestrator 710 receives a taskcomprising a labeling request with accompanying task information, suchas a task type description with associated constraints (e.g., targetconfidence threshold, cost constraints (temporal and/or monetary) orother constraints).

At step 1102, workflow orchestrator 710 applies criteria to filter outlabelers from consideration. Examples of filter criteria include, butare not limited to: task type description must match part or all of therequest task description exactly; descriptors, if specified, must matchaccording to Boolean logic specified for workflow configuration; poolsize must be greater than or equal to the number of times that labelerhas been called so far by the workflow orchestrator for the samelabeling request (i.e., there must be at least one more labeler instancein the labeler's pool that has not already provided a result for thelabeling request). If all the labelers are filtered out, as determinedat step 1106, workflow orchestrator 710 may output an exception (step1108).

If there are cost constraints (temporal, monetary or other cost) and/orconfidence constraints, workflow orchestrator 710 performs a path searchto find the optimal set of labelers (step 1114). For example, workfloworchestrator 710 determines a bounded number of candidate possibleexecution paths through the set of labelers. According to oneembodiment, paths are generated and evaluated using a configurable A* orbreadth first search with max depth, and an evaluation function isapplied. Other path search algorithms can also be used.

For each node in the search, the accrued cost and accrued resultconfidence estimate (based on the contributions from labelerconsultations in the path) are considered. For the first labeler in theexecution path through the CDW, there may be no accrued cost or accruedconfidence estimate. In other embodiments, an accrued cost and resultconfidence estimate may be provided to the CDW for a labeling request.

The evaluation function for each node in the search can be aconfigurable function of total path monetary cost, total path time, andend-point confidence (which can be estimated by QMS 750 for each pathendpoint). For a path, the temporal costs associated with each labelerin a path can be used in the evaluation function to determine theestimated total path time and the monetary costs associated with eachlabeler in the path can be used in the evaluation function to determinethe estimated total path monetary cost. As discussed above, the temporalcosts and monetary costs may be expressed as statistical aggregates(mean and variance, 95% confidence interval range or other statisticalaggregate). Thus, a statistical aggregate may be determined for the pathas a whole.

Similarly, when considering each labeler in the search, any statisticalaggregate (mean, median) or distribution (e.g., range or 95% confidenceinterval) can be used in the evaluation function to represent theconfidence estimate for that labeler's position in the sequence oflabelers consulted. For example, the labeler's score (which may be astatistical description derived from the scores of the labeler instancesin that labelers pool) may be used to represent a confidence estimatefor that labeler's position.

More particularly, each path endpoint can represent a specific path oflabelers (that is, a sequence of labelers consulted) and QMS 750estimates the a priori endpoint confidence for the path (step 1120)before the path is executed. QMS 750 estimates the a priori endpointconfidence based on the same confidence calculation it would use tocalculate a confidence estimate for a result once the path is executed.The difference in estimating the a priori confidence of the path beforethe path is executed and determining a confidence estimate after thepath is executed is that before, the workflow orchestrator uses thelabeler scores (e.g., the confidence ranges of the individual labelers)as confidence estimates for the labelers (e.g., uses the mean/median ormax/min, or other statistical description depending on how it isconfigured of the scores for the pool of instances) whereas after, theworkflow orchestrator uses information about the specific labelerinstance that performed the labeling. Thus, for example, QMS 750 mayapply various methods of determining complex granular confidences and/orcomposite confidences as described above, using the scores of theconstituent labelers to determine a confidence range for a path.

If the path meets the task constraints based on the estimated total pathmonetary cost, total path time, and end-point confidence, the path canbe added to a set of viable paths (step 1120). Otherwise the path can bediscarded (step 1122).

According to one embodiment, a path may be considered to meet the taskconstraints based on a set of configurable criteria. For example,according to one embodiment, a path may be considered to meet the taskconstraints if i) cost target, if specified, must be >=95% confidenceinterval determined for the path; ii) time target, if specified, mustbe >=95% confidence interval confidence determined for path; and iii)confidence target, if specified, must be within estimated confidencerange or confidence interval determined for the path. Other rules may beapplied. For example, the 95% confidence interval is merely provided asan example and other intervals may be used.

As illustrated in FIG. 12A, the path search can begin by evaluatingsingle nodes as the first path step (e.g., evaluate node 1200 toevaluate a single node path 1201). In this case, the estimated totalpath monetary cost may be equal to the monetary cost of ML labeler 712,the total path time cost may be equal to the time cost of ML labeler712, and the estimated end-point confidence may be equal to the score(e.g., confidence range) for ML labeler 712 for the task type for whichthe path is being determined. In one embodiment, path 1201 may beconsidered to meet the task constraints if i) cost target, if specified,is >=95% confidence interval determined for path 1201; ii) time target,if specified, is >=95% confidence interval confidence determined forpath 1201; and iii) confidence target, if specified, is within estimatedend-point confidence range or confidence interval determined for thepath 1201. If path 1201 meets the constraints, path 1201 can beconsidered to be a viable (candidate) path. If path 1201 does not meetthe constraints, path 1201 is not considered viable. Other rules may beapplied. For example, the 95% confidence interval is merely provided asan example and other intervals may be used.

The path search can continue to search for viable paths. As illustratedin FIG. 12B, for example, the path search can add a node 1202 for afirst consultation of human labeler 714 to create path 1203. In thiscase, QMS 750 estimates the total monetary cost of path 1203 byaggregating the monetary costs of ML labeler 712 and human labeler 714and estimates the total time cost of path 1203 by aggregating the timecosts of ML labeler 712 and human labeler 714. QMS 750 estimates thepath end-point confidence by applying the appropriate confidenceestimate calculation. For example, since human labeler 714 is a “blind”judgement labeler, QMS 750 could apply a blind granular confidenceestimate using the score (e.g., confidence ranges) of ML labeler 712 andhuman labeler 714 to determine a confidence range for the overall path1203. In one embodiment, path 1203 may be considered to meet the taskconstraints if i) cost target, if specified, is >=95% confidenceinterval determined for path 1203; ii) time target, if specified,is >=95% confidence interval confidence determined for path 1203; andiii) confidence target, if specified, is within estimated end-pointconfidence range or confidence interval determined for the path 1203. Ifpath 1203 meets the constraints, path 1203 can be considered to be aviable path. If path 1203 does not meet the constraints, path 1203 isnot considered viable. Other rules may be applied. For example, the 95%confidence interval is merely provided as an example and other intervalsmay be used.

FIG. 12C illustrates some additional examples of paths, including path1205 with human labeler 714 used twice (with different labelerinstances), path 1207, path 1209, path 1211, path 1215, and path 1217.

In some embodiments, the search can be configured to stop when a maxdepth or cost is reached without finding a path that meets theconstraints (FAILURE).

In some embodiments, the search can also stop when a single path isfound that meets the constraints. In yet another embodiment, the pathsearch can be stopped when some number of paths have been explored andworkflow orchestrator 710 can pick the optimal path from the possiblymultiple solutions found that meet the constraints.

If the search ends without finding a viable path, workflow orchestrator710 can output an exception (step 1126). If at least one viable path isfound, workflow orchestrator can select a path from the one or moreviable paths (step 1128). If there are multiple viable paths, thisselection may be random. In another embodiment, if there are one or morepaths that meet the overall constraints, workflow orchestrator canoptimize the search path according to various criteria. According to oneembodiment, workflow orchestrator can optimize the path according to oneor more of:

-   -   1) Cost optimized. At each branch in the search tree, select the        next node in the path as the labeler with the least mean        monetary cost, if there is a tie between labelers in a path        randomly select from among this subset;    -   2) Time optimized. At each branch in the search tree, select the        next node in the path as the labeler with the least mean time        cost, if there is a tie between labelers in a path randomly        select from among this subset;    -   3) Confidence optimized. At each branch in the search tree,        select the next node in the path as the labeler with the highest        minimum confidence estimate (e.g., the labeler with a score        range with the highest minimum score) or highest mean confidence        estimate (e.g., the labeler with the confidence range with the        highest minimum score);    -   4) Combinations of these can be used by optimizing a set of        confidence constraints for a range, then optimizing the subset        for a different range on a different variable. This can all be        specified in the configuration.

If the selected viable path includes only one labeler (one node) (asdetermined at step 1129), the minimum confidence required for thelabeler in order to maintain viability of the planned path may bedetermined (step 1130). For example, workflow orchestrator 710 can queryQMS 750 for the minimum confidence required for the labeler in order tomaintain viability of the planned path (may be subset of the confidencerange the labeler provides). According to one embodiment, QMS 750 backsolves the confidence estimate for the labeler to determine the minimumconfidence required. At step 1132, workflow orchestrator 710 sends thetask to that labeler with a confidence constraint, the confidenceconstraint including the required minimum confidence determined by QMS750.

At any point, if all remaining constraints and optimizations aresatisfied by multiple labelers, then a random selection is made fromthese labelers. Workflow orchestrator 710 selects the first labeler in aselected path (step 1134) and sends the task to that labeler with orwithout a confidence constraint (step 1136).

At steps 1132 and 1136, workflow orchestrator 710 sends the selectedlabeler a labeling request, an instance identifier list for the labelerinstances behind that labeler that have already executed the samelabeling request (and therefore may be excluded from providing anotheranswer), and a set of constraints, such as, but not limited to, a simpleconfidence estimate constraint (provided by the QMS 750 or from searchpath constraints), and a maximum time and cost to completion (from thetask information or from search path constraints based on the value usedto calculate total path time for a node). The constraints areconceptually an “SLA contract” for maximum time, maximum price, andminimum confidence allowed for the labeler. One embodiment of processingby a labeler is illustrated in FIG. 13 .

At step 1138, workflow orchestrator 710 receives an output of thelabeler. Workflow orchestrator collects labeled results corresponding toa task (or labeling request) (as illustrated by constituent labelerresults 720) (step 1140). If the labeler returns an exception, workfloworchestrator 710 may implement exception handling (step 1141). In someembodiments, workflow orchestrator 710 outputs the exception andprocessing of the input request may end. In other embodiments, workfloworchestrator may continue processing the input request (e.g., by routingthe labeling request to a next labeler).

According to one embodiment, workflow orchestrator 710 determines theconfidence estimate for the label returned by the labeler (step 1142).In a particular embodiment, workflow orchestrator 710 calls the QMS 750to calculate the confidence estimate for the label and QMS 750 uses, forexample, a granular confidence approach. If the input request hasalready been processed by a labeler in the path, QMS 750 may use acomplex granular confidence approach (e.g. probability formulas orlogistic regression) based on the labeled results output by the labelerand prior labelers, previous accuracy assessments for the labelerinstances, labeler instance scores, and/or self-reported confidence forall labeler instances consulted in the path thus far.

The workflow orchestrator 710 examines the result. Note that even if anindividual labeler confidence constraint is provided to a labeler, theQMS-assessed confidence in the returned result may be lower than theindividual labeler's a priori confidence estimate and lower than theconstraint provided. For example, this may happen for ambiguous inputswhen the labeler's (self-reported) certainty is taken into account forconfidence estimates.

If the confidence estimate returned for the label by QMS 750 meets theconfidence threshold target, as determined at step 1144, then a stoppingcondition has been reached, and workflow orchestrator 710 can return thelabeled result, including the confidence determined by QMS 750 for thelabel (step 1146).

If the confidence estimate returned by QMS 750 does not meet theconfidence threshold, workflow orchestrator 710 may determine ifadditional routing is available (step 1148). In some embodiments,workflow orchestrator 710 continues processing the labeling requestusing the path selected at step 1128 until the target confidencethreshold is met or the path is exhausted. If the path is not complete,workflow orchestrator 710 routes the labeling request to the nextlabeler (step 1150). For example, if workflow orchestrator 710 selectspath 1211 for processing a labeling request, workflow orchestrator 710will continue routing the labeling request using path 1211 until theconfidence threshold target is met or path 1211 is exhausted. If thereare no remaining labelers in the selected path, workflow orchestrator710 will output an exception (step 1152).

In another embodiment, workflow orchestrator 710 re-searches forpossible paths after consultation with the prior labeler and therebydetermines a new set of viable paths, each of which starts with thelabelers consulted so far. The possible paths may be searched againusing the remaining labelers in CDW 700. At this point, workfloworchestrator has more information about the task, the answers alreadygiven by labelers in the execution path, the confidence estimates inthose answers, and the costs accrued by each labeler in processing thelabeling request. Workflow orchestrator 710 can thus query QMS forscores conditioned on that answer, and determine viable paths given theaccrued costs and confidence estimates in the answers so far. Thus,workflow orchestrator 710 can route low-confidence “results so far”differently from high-confidence “results so far”.

For example, say workflow orchestrator 710 initially selects path 1201,but the confidence estimate for the result output by ML labeler 712 doesnot meet the threshold confidence, workflow orchestrator can perform apath search to find viable paths given the actual costs accrued by theprior nodes in the execution path—ML labeler 712 in this example—and theconfidence estimate for the labeled result output by the prior node. Asanother example, say step 1148 occurs after executing the labelingrequest according to path 1203, then workflow orchestrator can perform apath search to find viable paths given the actual costs accrued by theprior nodes in the execution path—ML labeler 712 and ML labeler 712 inthis example—and the confidence estimate for the labeled results outputby ML labeler 712 and human labeler 714.

Thus, in performing steps 1148 and 1150, workflow orchestrator 710 canessentially repeat steps 1104-1136 starting from the last node used inthe path so far and considered the accrued costs and result confidenceestimate based on one or more prior labeler consultations. If the searchfails (e.g., due to exceeded time/cost constraints or failed escalatingconfidence requirements (for ambiguous cases)), workflow orchestrator710 will output an exception (step 1154).

In some (early bootstrapping, crowd training) workflow configurationswhere, for example, the confidence models are not yet stable for givenlabelers, if the assessed confidence by QMS 750 in the result returnedby the labeler is lower than the target specified by the workfloworchestrator 710 (by some configurable threshold), then the workfloworchestrator 710 can be configured to throw away the result and retrywith a different labeler instance (escalate confidence constraints andthen timeout with exception based on configurable number of retries).

FIG. 11A and FIG. 11B are provided by of example and not limitation.Various steps may be repeated, steps may be performed in differentorders, steps omitted, and additional or alternative steps performed.

FIG. 13 is a flow chart illustrating one embodiment a method of alabeler processing a task (labeling request). The steps of FIG. 13 maybe embodied as computer-executable instructions on a non-transitorycomputer-readable medium. One or more steps of FIG. 13 may beimplemented by a labeling platform. By way of example, but notlimitation, one or more steps of FIG. 13 may be performed by a labeler.

At step 1302, a labeler (e.g., labeler 712, 714, 716) receives a taskfrom workflow orchestrator 710. The task may include cost (e.g., maximumtime, maximum price) or confidence constraints (e.g., an individuallabeler confidence target). If cost constraints are provided, thelabeler determines if it can complete the task in the cost constraints(step 1304) and, if not, returns an exception (step 1306). In someembodiments, all cost estimates in a particular cost dimension (e.g.,monetary, time) for labeler instances within a single labeler areuniform (e.g., each labeler instance behind a labeler is assumed to havethe same monetary cost estimate and each labeler instance behind alabeler is assumed to have the same time cost estimate). In otherembodiments, different labeler instances may have different associatedcosts and the labeler filters out labeler instances that are estimatedto not meet the cost constraints. If there are no remaining labelerinstances to handle the task, the labeler returns an exception (step1306).

If a list of labeler instances in the labeler's pool that have processedthe labeling request is provided, the labeler filters out the labelerinstances that are excluded from processing the task based on thelabeler instance list (step 1308). If there are no remaining labelerinstances to handle the task, the labeler returns an exception (step1310).

If an individual labeler confidence constraint is provided, the labelerdetermines, from the remaining labeler instances, a subset of labelerinstances that are estimated to meet the individual labeler confidenceconstraint (step 1312). As will be appreciated from the discussionabove, the selected path of which the labeler is a part is calculated tomake the overall label result confidence meet the configured confidencethreshold. An individual labeler is selected to be a part of the pathbased on its anticipated contribution to that overall confidence. Itsanticipated impact is based on quality metrics provided by QMS 750.

As the path is executed, the confidence constraint on the remaininglabelers will be updated to reflect how much confidence the next (andsubsequent) labelers have to provide to get the overall confidence adesired value. If the first labeler provides a very confident answer,then less is needed from the second to get the overall confidence to theright point. The individual labeler confidence constraint is specific towhat is needed from that labeler for its part of the path and iscalculated by the QMS 750. The individual labeler confidence constraintcan be represented as a simple confidence estimate target.

According to one embodiment, the labeler queries QMS 750 for the subsetof labeler instances of that labeler that have a priori confidenceestimates for the label that meet the confidence constraint for thelabeler's part in the overall path (or for larger pools, a random subsetof n labeler instances that are predicted to meet the confidenceconstraint). If there is no labeler instance that meets the individuallabeler confidence constraint, the labeler may return an exception (step1314).

At step 1316, the labeler selects a labeler instance for the subset oflabeler instances that are estimated meet the constraints to handle thetask. According to one embodiment, this selection is random. In anotherembodiment, for example, the labeler selects the labeler instance basedon load-balancing concerns.

The labeler sends the task to the labeler instance (step 1318) andreceives the result from the labeler instance (step 1320). The labelerreturns the result provided by the labeler instance to workfloworchestrator 710 (including the info about which labeler instance didthe task, any self-assessed confidence provided by the labeler instance)(step 1322).

FIG. 13 provided by of example and not limitation. Various steps may berepeated, steps may be performed in different orders, steps omitted, andadditional or alternative steps performed.

FIG. 14 is a diagrammatic representation of one embodiment of adistributed network computing environment where embodiments disclosedherein can be implemented. The computing environment includes a labelingplatform system 1400, one or more second computer systems 1430 connectedto a network 1405 (e.g., a local area network (LAN), a wide area network(WAN) such as the Internet, mobile network, or other type of network orcombination thereof). Network 1405 may represent a combination of wiredand wireless networks that network computing environment may utilize forvarious types of network communications.

Labeling platform system 1400 may include, for example, a computerprocessor 1402 and associated memory 1404. Computer processor 1402 maybe an integrated circuit for processing instructions. For example,processor 1402 may comprise one or more cores or micro-cores of aprocessor. Memory 1404 may include volatile memory, non-volatile memory,semi-volatile memory, or a combination thereof. Memory 1404, forexample, may include RAM, ROM, flash memory, a hard disk drive, asolid-state drive, an optical storage medium (e.g., CD-ROM), or othercomputer readable memory or combination thereof. Memory 1404 mayimplement a storage hierarchy that includes cache memory, primarymemory, or secondary memory. In some embodiments, memory 1404 mayinclude storage space on a data storage array. Labeling platform system1400 may also include input/output (“I/O”) devices 1406, such as akeyboard, monitor, printer, electronic pointing device (e.g., mouse,trackball, stylus, etc.), or the like. Labeling platform system 1400 mayalso include a communication interface 1410, such as a network interfacecard, to interface with network 1405.

Memory 1404 may store instructions executable by processor 1402. Forexample, memory 1404 may include instructions executable to implement alabeling platform, such as labeling platform 104. In some embodiments,memory 1404 may include instructions to implement a confidence drivenworkflow and QMS. Labeling platform system 1400 may represent aplurality of servers. In some embodiments, labeling platform system 1400may represent a cloud computing system.

Labeling platform system 1400 may include a data store 1420 that storesdata usable by the labeling platform. According to one embodiment, datastore 1420 may comprise one or more databases, one or more file systemsor a combination thereof. In some embodiments, data store 1420 may be aportion of memory 1404.

Second computer system 1430 may include, for example, a computerprocessor 1432 and associated memory 1434. Computer processor 1432 maybe an integrated circuit for processing instructions. For example,processor 1432 may comprise one or more cores or micro-cores of aprocessor. Memory 1434 may include volatile memory, non-volatile memory,semi-volatile memory, or a combination thereof. Memory 1434, forexample, may include RAM, ROM, flash memory, a hard disk drive, asolid-state drive, an optical storage medium (e.g., CD-ROM), or othercomputer readable memory or combination thereof. Memory 1434 mayimplement a storage hierarchy that includes cache memory, primarymemory, or secondary memory. In some embodiments, memory 1434 mayinclude storage space on a data storage array. Second computer system1430 may also I/O devices 1436. Second computer system 1430 may alsoinclude a communication interface 1440, such as a network interfacecard, to interface with network 1405.

Memory 1434 may store instructions executable by processor 1432. Forexample, memory 1434 may include one or more programs to implement ahuman labeler computer system 140 or a client computer system 150. Whileonly one computer system 1430 is illustrated, there may be a largenumber of second computer systems 1430 connected to labeling platformsystem 1400.

Labeling platform system 1400 may also be coupled to an ML platformsystem 1450. ML platform system 1450 may include, for example, acomputer processor 1452 and associated memory 1454. Computer processor1452 may be an integrated circuit for processing instructions. Forexample, processor 1452 may comprise one or more cores or micro-cores ofa processor. Memory 1454 may include volatile memory, non-volatilememory, semi-volatile memory, or a combination thereof. Memory 1454, forexample, may include RAM, ROM, flash memory, a hard disk drive, asolid-state drive, an optical storage medium (e.g., CD-ROM), or othercomputer readable memory or combination thereof. Memory 1454 mayimplement a storage hierarchy that includes cache memory, primarymemory, or secondary memory. In some embodiments, memory 1454 mayinclude storage space on a data storage array. ML platform system 1450may also include input/output (“I/O”) devices 1456, such as a keyboard,monitor, printer, electronic pointing device (e.g., mouse, trackball,stylus, etc.), or the like. ML platform system 1450 may also include acommunication interface 1460, such as a network interface card, tointerface with network 1405.

Memory 1454 may store instructions executable by processor 1452. Forexample, memory 1454 may include instructions executable to implement anML model platform that allow for the training or deployment of MLmodels. ML platform system 1450 may represent a plurality of servers. Insome embodiments, ML platform system 1450 may represent a cloudcomputing system. While only one ML platform system 1450 is illustrated,labeling platform system may utilize any number of ML platform systems.

Although the invention has been described with respect to specificembodiments thereof, these embodiments are merely illustrative, and notrestrictive of the invention. The description herein is not intended tobe exhaustive or to limit the invention to the precise forms disclosedherein (and in particular, the inclusion of any particular embodiment,feature or function is not intended to limit the scope of the inventionto such embodiment, feature or function). Rather, the description isintended to describe illustrative embodiments, features and functions inorder to provide a person of ordinary skill in the art context tounderstand the invention without limiting the invention to anyparticularly described embodiment, feature or function. While specificembodiments of, and examples for, the invention are described herein forillustrative purposes only, various equivalent modifications arepossible within the spirit and scope of the invention, as those skilledin the relevant art will recognize and appreciate. As indicated, thesemodifications may be made to the invention in light of the foregoingdescription of illustrated embodiments of the invention and are to beincluded within the spirit and scope of the invention.

Thus, while the invention has been described herein with reference toparticular embodiments thereof, a latitude of modification, variouschanges and substitutions are intended in the foregoing disclosures, andit will be appreciated that in some instances some features ofembodiments of the invention will be employed without a correspondinguse of other features without departing from the scope and spirit of theinvention as set forth. Therefore, many modifications may be made toadapt a particular situation or material to the essential scope andspirit of the invention.

Reference throughout this specification to “one embodiment”, “anembodiment”, or “a specific embodiment” or similar terminology meansthat a particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodimentand may not necessarily be present in all embodiments. Thus, respectiveappearances of the phrases “in one embodiment”, “in an embodiment”, or“in a specific embodiment” or similar terminology in various placesthroughout this specification are not necessarily referring to the sameembodiment. Furthermore, the particular features, structures, orcharacteristics of any particular embodiment may be combined in anysuitable manner with one or more other embodiments. It is to beunderstood that other variations and modifications of the embodimentsdescribed and illustrated herein are possible in light of the teachingsherein and are to be considered as part of the spirit and scope of theinvention.

Additionally, any examples or illustrations given herein are not to beregarded in any way as restrictions on, limits to, or expressdefinitions of, any term or terms with which they are utilized. Instead,these examples or illustrations are to be regarded as being describedwith respect to one particular embodiment and as illustrative only.Those of ordinary skill in the art will appreciate that any term orterms with which these examples or illustrations are utilized willencompass other embodiments which may or may not be given therewith orelsewhere in the specification and all such embodiments are intended tobe included within the scope of that term or terms. Language designatingsuch nonlimiting examples and illustrations includes, but is not limitedto: “for example,” “for instance,” “e.g.,” “in one embodiment.”

In the description herein, numerous specific details are provided, suchas examples of components and/or methods, to provide a thoroughunderstanding of embodiments of the invention. One skilled in therelevant art will recognize, however, that an embodiment may be able tobe practiced without one or more of the specific details, or with otherapparatus, systems, assemblies, methods, components, materials, parts,and/or the like. In other instances, well-known structures, components,systems, materials, or operations are not specifically shown ordescribed in detail to avoid obscuring aspects of embodiments of theinvention. While the invention may be illustrated by using a particularembodiment, this is not and does not limit the invention to anyparticular embodiment and a person of ordinary skill in the art willrecognize that additional embodiments are readily understandable and area part of this invention.

Those skilled in the relevant art will appreciate that embodiments canbe implemented or practiced in a variety of computer systemconfigurations including, without limitation, multi-processor systems,network devices, mini-computers, mainframe computers, data processors,and the like. Embodiments can be employed in distributed computingenvironments, where tasks or modules are performed by remote processingdevices, which are linked through a communications network such as aLAN, WAN, and/or the Internet. In a distributed computing environment,program modules or subroutines may be located in both local and remotememory storage devices. These program modules or subroutines may, forexample, be stored or distributed on computer-readable media, stored asfirmware in chips, as well as distributed electronically over theInternet or over other networks (including wireless networks). Examplechips may include Electrically Erasable Programmable Read-Only Memory(EEPROM) chips.

Embodiments described herein can be implemented in the form of controllogic in software or hardware or a combination of both. The controllogic may be stored in an information storage medium, such as acomputer-readable medium, as a plurality of instructions adapted todirect an information processing device to perform a set of stepsdisclosed in the various embodiments. Based on the disclosure andteachings provided herein, a person of ordinary skill in the art willappreciate other ways and/or methods to implement the invention. Steps,operations, methods, routines or portions thereof described herein beimplemented using a variety of hardware, such as CPUs, applicationspecific integrated circuits, programmable logic devices, fieldprogrammable gate arrays, optical, chemical, biological, quantum ornanoengineered systems, or other mechanisms.

Software instructions in the form of computer-readable program code maybe stored, in whole or in part, temporarily or permanently, on anon-transitory computer readable medium. The computer-readable programcode can be operated on by a processor to perform steps, operations,methods, routines or portions thereof described herein. A“computer-readable medium” is a medium capable of storing data in aformat readable by a computer and can include any type of data storagemedium that can be read by a processor. Examples of non-transitorycomputer-readable media can include, but are not limited to, volatileand non-volatile computer memories, such as RAM, ROM, hard drives, solidstate drives, data cartridges, magnetic tapes, floppy diskettes, flashmemory drives, optical data storage devices, compact-disc read-onlymemories. In some embodiments, computer-readable instructions or datamay reside in a data array, such as a direct attach array or otherarray. The computer-readable instructions may be executable by aprocessor to implement embodiments of the technology or portionsthereof.

A “processor” includes any, hardware system, mechanism or component thatprocesses data, signals or other information. A processor can include asystem with a general-purpose central processing unit, multipleprocessing units, dedicated circuitry for achieving functionality, orother systems. Processing need not be limited to a geographic location,or have temporal limitations. For example, a processor can perform itsfunctions in “real-time,” “offline,” in a “batch mode,” etc. Portions ofprocessing can be performed at different times and at differentlocations, by different (or the same) processing systems.

Different programming techniques can be employed such as procedural orobject oriented. Any suitable programming language can be used toimplement the routines, methods or programs of embodiments of theinvention described herein, including R, Python, C, C++, Java,JavaScript, HTML, or any other programming or scripting code, etc.Communications between computers implementing embodiments can beaccomplished using any electronic, optical, radio frequency signals, orother suitable methods and tools of communication in compliance withknown network protocols. Any particular routine can execute on a singlecomputer processing device or multiple computer processing devices, asingle computer processor or multiple computer processors. Data may bestored in a single storage medium or distributed through multiplestorage mediums. In some embodiments, data may be stored in multipledatabases, multiple filesystems, or a combination thereof.

Although the steps, operations, or computations may be presented in aspecific order, this order may be changed in different embodiments. Insome embodiments, some steps may be omitted. Further, in someembodiments, additional or alternative steps may be performed. In someembodiments, to the extent multiple steps are shown as sequential inthis specification, some combination of such steps in alternativeembodiments may be performed at the same time. The sequence ofoperations described herein can be interrupted, suspended, or otherwisecontrolled by another process, such as an operating system, kernel, etc.The routines can operate in an operating system environment or asstand-alone routines. Functions, routines, methods, steps and operationsdescribed herein can be performed in hardware, software, firmware or anycombination thereof.

It will be appreciated that one or more of the elements depicted in thedrawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application.Additionally, any signal arrows in the drawings/figures should beconsidered only as exemplary, and not limiting, unless otherwisespecifically noted.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having,” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,product, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, product, article, orapparatus.

Furthermore, the term “or” as used herein is generally intended to mean“and/or” unless otherwise indicated. For example, a condition A or B issatisfied by any one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present). As used herein, a termpreceded by “a” or “an” (and “the” when antecedent basis is “a” or “an”)includes both singular and plural of such term, unless clearly indicatedwithin the claim otherwise (i.e., that the reference “a” or “an” clearlyindicates only the singular or only the plural). Also, as used in thedescription herein and throughout the meaning of “in” includes “in” and“on” unless the context clearly dictates otherwise.

Although the foregoing specification describes specific embodiments,numerous changes in the details of the embodiments disclosed herein andadditional embodiments will be apparent to, and may be made by, personsof ordinary skill in the art having reference to this disclosure. Inthis context, the specification and figures are to be regarded in anillustrative rather than a restrictive sense, and all such modificationsare intended to be included within the scope of this disclosure.

What is claimed is:
 1. A computer program product comprising anon-transitory, computer-readable medium storing thereon a set ofcomputer-executable instructions, the set of computer-executableinstructions comprising instructions for: providing a confidence-drivenworkflow (CDW) comprising a set of labelers, each labeler in the set oflabelers comprising executable code and having a dynamically modeledconfidence range; dynamically determining an execution path forprocessing a labeling request to label a data item, wherein dynamicallydetermining the execution path comprises: dynamically determining abounded number of candidate paths through the set of labelers usingdynamically calculated cost and confidence metrics for the labelers inthe set of labelers to estimate a probability of each candidate path tosatisfy a set of constraints on cost and final result confidence;selecting a candidate path from the candidate paths as a selected path;and executing a next labeler consultation according to the selected pathto label the data item.
 2. The computer program product of claim 1,wherein the set of computer-executable instructions further comprisesinstructions for: continually monitoring and scoring a plurality oflabeler instances to generate labeler instance scores for the pluralityof labeler instances, wherein scoring a labeler instance comprisesdetermining an accuracy of the labeler instance based on a correctnessof a set labeled results produced by the labeler instance; and updatingthe dynamically modeled confidence range of each labeler from the set oflabelers, wherein updating the dynamically modeled confidence range foreach labeler from set of labelers comprises: determining a set oflabeler instance scores associated with a pool of labeler instancesrepresented by each labeler from the set of labelers; and for eachlabeler from the set of labelers, aggregating the set of labelerinstance scores associated with the pool of labeler instancesrepresented by that labeler to generate the dynamically modeledconfidence range for that labeler.
 3. The computer program product ofclaim 1, wherein each candidate path from the bounded number ofcandidate paths has an associated path cost and an associated pathconfidence and wherein the selected path is selected to optimize theassociated path cost and meet a target confidence threshold.
 4. Thecomputer program product of claim 1, wherein the bounded number ofcandidate paths through the set of labelers is determined based on anaccrued cost and a result confidence estimate based on one or more priorlabeler consultations in the execution path.
 5. The computer programproduct of claim 1, wherein the set of computer-executable instructionsfurther comprise instructions for: receiving a labeled result based onexecuting the next labeler consultation; and determining a confidenceestimate for the labeled result.
 6. The computer program product ofclaim 5, wherein the set of computer-executable instructions furthercomprise instructions for: based on a determination that the confidenceestimate for the labeled result meets a target confidence threshold,outputting the labeled result as a final result for the CDW.
 7. Thecomputer program product of claim 6, wherein the set ofcomputer-executable instructions further comprise instructions for:based on a determination that the confidence estimate for the labeledresult does not meet the target confidence threshold, performing atleast one of: routing the labeling request to a next labeler from theset of labelers according to the selected path; or reporting anexception that the labeling request cannot be completed within the setof constraints on cost and final result confidence.
 8. The computerprogram product of claim 6, wherein the set of computer-executableinstructions further comprise instructions for: based on a determinationthat the confidence estimate for the labeled result does not meet thetarget confidence threshold: dynamically determining a new set ofcandidate paths; selecting a first candidate path from the new set ofcandidate paths as a new selected path based on the first candidate pathminimizing a total path cost; and routing the labeling request to a nextlabeler from the set of labelers according to the new selected path. 9.The computer program product of claim 1, wherein each labeler in the setof labelers represents one or more labeler instances from a plurality oflabeler instances and wherein the set of computer-executableinstructions further comprises instructions for: determining for eachlabeler instance in the plurality of labeler instances a score for thatlabeler instance that corresponds to a probability that the labelerinstance will produce an accurate label; and determining the dynamicallymodeled confidence range for each labeler in the set of labelers as astatistical description derived from the scores of the one or morelabeler instances represented by that labeler.
 10. The computer programproduct of claim 9, wherein each labeler instance in the plurality oflabeler instances has an associated labeler instance cost and whereineach labeler has an associated labeler cost, wherein the associatedlabeler cost for each labeler is a statistical description of theassociated labeler instance costs of the one or more labeler instancesrepresented by that labeler.
 11. The computer program product of claim9, wherein the set of computer-executable instructions further comprisesinstructions for determining confidence estimates for labeled resultsproduced by each labeler of the set of labelers based on the scoresdetermined for the one or more labeler instances represented by thatlabeler.
 12. The computer program product of claim 11, wherein at leastone labeler in the set of labelers comprises instructions for: receivingan individual labeler confidence constraint for the labeling request;determining that a labeler instance of the one or more labeler instancesrepresented by at least one labeler has a score that meets theindividual labeler confidence constraint; and routing the labelingrequest to the labeler instance.
 13. A computer-implemented method fordata labeling: providing a confidence-driven workflow (CDW) comprising aset of labelers, each labeler in the set of labelers comprisingexecutable code and having a dynamically modeled confidence range;dynamically determining an execution path for processing a labelingrequest to label a data item, wherein dynamically determining theexecution path comprises: dynamically determining a bounded number ofcandidate paths through the set of labelers using dynamically calculatedcost and confidence metrics for the labelers in the set of labelers toestimate a probability of each candidate path to satisfy a set ofconstraints on cost and final result confidence; selecting a candidatepath from the candidate paths as a selected path; and executing a nextlabeler consultation according to the selected path to label the dataitem.
 14. The computer-implemented method of claim 13, furthercomprising continually monitoring and scoring a plurality of labelerinstances to generate labeler instance scores for the plurality oflabeler instances, wherein scoring a labeler instance comprisesdetermining an accuracy of the labeler instance based on a correctnessof a set labeled results produced by the labeler instance; and updatingthe dynamically modeled confidence range of each labeler from the set oflabelers, wherein updating the dynamically modeled confidence range ofeach labeler from the set of labelers comprises: determining a set oflabeler instance scores associated with a pool of labeler instancesrepresented by each labeler from the set of labelers; and for eachlabeler from the set of labelers, aggregating the set of labelerinstance scores associated with the pool of labeler instancesrepresented by that labeler to generate the dynamically modeledconfidence range for that labeler.
 15. The computer-implemented methodof claim 13, wherein each candidate path from the bounded number ofcandidate paths has an associated path cost and an associated pathconfidence and wherein the selected path is selected to optimize theassociated path cost and meet a target confidence threshold.
 16. Thecomputer-implemented method of claim 13, wherein the bounded number ofcandidate paths through the set of labelers is determined based on anaccrued cost and a result confidence estimate based on one or more priorlabeler consultations in the execution path.
 17. Thecomputer-implemented method of claim 13, further comprising: receiving alabeled result based on executing the next labeler consultation; anddetermining a confidence estimate for the labeled result.
 18. Thecomputer-implemented method of claim 17, further comprising: based on adetermination that the confidence estimate for the labeled result meetsa target confidence threshold, outputting the labeled result as a finalresult for the CDW.
 19. The computer-implemented method of claim 17,further comprising: based on a determination that the confidenceestimate for the labeled result does not meet a target confidencethreshold, performing at least one of: routing the labeling request to anext labeler from the set of labelers according to the selected path; orreporting an exception that the labeling request cannot be completedwithin the a set of constraints on cost and final result confidence. 20.The computer-implemented method of claim 17, further comprising: basedon a determination that the confidence estimate for the labeled resultdoes not meet a target confidence threshold: dynamically determining anew set of candidate paths; selecting a first candidate path from thenew set of candidate paths as a new selected path based on the firstcandidate path minimizing a total path cost; and routing the labelingrequest to a next labeler from the set of labelers according to the newselected path.
 21. The computer-implemented method of claim 13, furthercomprising: for each labeler instance in a plurality of labelerinstances, determining a score for that labeler instance thatcorresponds to a probability that the labeler instance will produce anaccurate label, wherein each labeler in the set of labelers representsone or more labeler instances from the plurality of labeler instances;and determining the dynamically modeled confidence range for eachlabeler in the set of labelers as a statistical description derived fromthe scores of the one or more labeler instances represented by thatlabeler.
 22. The computer-implemented method of claim 21, wherein eachlabeler instance in the plurality of labeler instances has an associatedlabeler instance cost and wherein each labeler has an associated labelercost, wherein the associated labeler cost for each labeler is astatistical description of the associated labeler instance costs of theone or more labeler instances represented by that labeler.
 23. Thecomputer-implemented method of claim 21, further comprising determiningconfidence estimates for labeled results produced by each labeler of theset of labelers based on the scores determined for the one or morelabeler instances represented by that labeler.
 24. Thecomputer-implemented method of claim 23, further comprising a selectedlabeler from the set of labelers performing: receiving an individuallabeler confidence constraint for the labeling request; determining thata labeler instance of the one or more labeler instances represented bythe selected labeler has a score that meets the individual labelerconfidence constraint; and routing the labeling request to the labelerinstance.